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CHAPTER 1 

INTRODUCTION 


INTRODUCTION 


Overview of the Study 

In recent years, interest has spiked in data-driven decision making in education— that is, using various types of data, par- 
ticularly quantitative assessment data, to inform a range of decisions in schools and classrooms (M arsh, Pane, & Hamilton, 
2006). This is a natural result of technological changes, the advent of test- based accountability systems under No Child Left 
Behind, and the increased availability of quantitative data due to accountability reforms. The increased emphasis on using 
data is based on the belief that assessment data and other student performance data can be important levers for improved 
teaching and learning. M any schools, districts, and states have invested resources in tools designed to provide teachers, 
principals, and other key stakeholders with ready access to (and analysis of) information regarding student performance 
throughout the school year. 

Of particular interest is the development of interim (also known as benchmark) assessments that are often adopted at the 
district level and are administered at regular intervals throughout the academic year. These assessments are intended to help 
teachers monitor and improve student learning, both in general and on the high-stakes, end-of-year accountability tests. 
There is a growing body of research on interim assessments and how they relate to data-driven instruction and decision mak- 
ing. Researchers also have examined the implementation of data practices in school districts and schools that are purport- 
edly making strides in data-driven decision making and instruction or that have undertaken significant initiatives in this area 
(e.g., Datnow, Park, & Wohlsetter, 2007; Snipes, Doolittle, & Herlihy, 2002). However, the field has yet to produce reliable 
evidence regarding the relationship between data use and teacher or school effectiveness at raising student achievement. 
A Ithough the literature includes case studies regarding views about and the use of interim assessments i n a particular school 
district, relatively few studies have attempted to generate specific estimates of the relationship between teacher data-use 
practices and perceptions and student achievement on end-of-year accountability tests. 

In October 2008, the Council of the Great City Schools and American Institutes for Research (AIR) launched a project 
funded by The Bill & M elinda Gates Foundation that focused on understanding the use of interim assessment data as a le- 
ver for instructional improvement. The study was conducted in four urban districts located in geographically distinct areas. 
The project had two interrelated objectives: (1) to document and understand current data-use practices across urban school 
districts in terms of the use and availability of data-in particular, the administration and use of interim assessments— and 
(2) to generate empirical evidence regarding the relationships between student achievement and data-use practices at the 
school and classroom levels. To address the first objective, we administered surveys to district academic/curriculum coor- 
dinators and research directors to obtain a general overview of the state of current practices in using data to inform school- 
and classroom-level decision making across urban school districts. Following the surveys, we conducted a series of case 
studies of four urban districts, allowing fora more in-depth look at district data use. For more information on the site visits, 
please seethe published report, titled Using Data to Improve Instruction in the Great City Schools: Documenting Current 
Practices, availableat www.cqcs.org . 
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This report focuses on the second objective: examining the empirical relationships between teacher and principal use of 
student interim assessment data and achievement on end-of-year accountability tests. I n this report, we expand on the ex- 
isting body of literature on the use of interim assessments by examining the extent to which data-use practices (including 
perceptions about using data) are related to student achievement. The report is organized in three sections. First, we review 
the literature on using data from interim assessments and put forth a theory of action that undergirds our investigation. The 
theory of action identifies a set of key dimensions of data use and hypothesizes that supporting conditions in states, school 
districts, and schools can facilitate the effective classroom-level use of data to respond to students' instructional needs. Sec- 
ond, we report results of an empirical test of this theory of action. The analyses examined the relationship between teachers' 
and schools' use of interim assessment data and improvements in student achievement in reading and mathematics at grades 
4, 5, 7, and 8. We focused on classroom-level data-use practices as reported by teachers, and school-level data-use practices 
and perceptions as reported by principals. The analyses include more than 1,500 teachers and 150 school principals and 
student achievement data from over 60,000 students across four urban districts collected during the 2009-10 school year. 
In the third and final section of this report, we interpret the study findings and provide recommendations and conclusions. 

Policy C ontext for the U se of Student Achievement Data 

Test scores have been used for some time to make instructional decisions, but test-score data have not always been easily 
available or systematically used to inform such decisions (Abelman, Elmore, Even, Kenyon, & M arshall, 1999). Recent 
efforts to implement systematic assessments at regular intervals during the school year hold promise for higher student 
achievement, and some researchers and practitioners suggest that their use may be critical to school improvement. Accord- 
ing to M arshall (2006), many schools that serve disadvantaged students who academically "beat the odds” analyze their 
interim assessment data as part of their overall strategy for improving achievement. I ndeed, studies that have examined the 
characteristics of high-performing schools and school districts havefound that data-driven instruction and decision making 
are common features in many of these organizations (Datnow etal., 2007; Snipes et al ., 2002). 

The continued advancement of technology and the growing pressure for schools to be data driven have resulted in sub- 
stantial new funding and research on educational data systems (Hamilton, 2005). With the U.S. Department of Education's 
desire to close achievement gaps through data use, new policies have been implemented to promote data use in schools 
and classrooms. The American Recovery and Reinvestment Act of 2009 called on states, school districts, and schools to 
develop longitudinal data systems to increase their capacity to support students' strengths and identify their weaknesses. 
This legislation sends a strong message about the importance of using data to inform educational practices and will inform a 
dialogue among multiple stakeholders on how data should and can be used in the future to improve public education. Other 
initiatives, such as the Data Quality Campaign (2009), have focused attention and resources on building state longitudinal 
databases that house student-level information for use by stakeholders at all levels. At the same time, access to student data 
is clearly growing. According to a nationally representative survey, for example, teacher access to student data systems grew 
from 48 percent in 2005 to 74 percent in 2007 (U.S. Department of Education, 2009). 

Creating the infrastructure to facilitate data-driven instruction has been a federal priority for several years. Since 2005, the 
State Longitudinal Data System (SLDS) grant program has distributed grants to assist states in developing data systems. In 
total, 74 grants have been awarded to 42 states. Of note, three of the districts participating in the study reported here were in 
states that received SLDS grants. These states areNevada, which received a grant in 2007; Kentucky, which received grants 
in 2005 and 2007; and Virginia, which received grants in 2007 and 2010. 
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The focus on data systems continued with the U.S. Department of Education's Race to the Top program. Race to the Top 
grants required states to describe both their current state data systems and their plans to improve those systems to a common 
standard. States applying for Race to the Top grants were expected to develop data systems that encompass elements of the 
AmericaCOMPETESAct. 1 Among these elements are a unique student identifier; a teacher identifier system with the abil- 
ity to match individual teachers to individual students; student-level enrollment, demographics, and participation; student- 
level information about points at which a student exits, transfers in or out, drops out, or completes PK - 16; the capacity to 
communicate with higher education systems; yearly state assessment records of individual students; student-level transcript 
information, including course completion and grade earned; student-level college readiness test scores; data on student tran- 
sition from secondary to postsecondary, including remedial coursework enrollment; and data necessary to address alignment 
and adequate preparation for success in postsecondary education and the workforce (America COM PETES Act, 2007). 

Despite the recent attention and investment in data systems at national, state, and local levels, researchers and practitioners 
have not reached consensus on what being data driven actually means in practice. M oreover, little evidence connects spe- 
cific data uses to changes in teaching and actual improvements in student outcomes. 

Literature and Previous Research on Assessment Data Use 

In this section of the report, we begin by reviewing different types of student assessments available to school districts, 
schools, and teachers. We next summarize the research on using data, particularly interim assessment data, to guide instruc- 
tional decisions and improve student outcomes. 

The literature on the use of data to inform instruction deals with various types of assessments and assessment strategies, 
including interim assessments, formative assessment, progress monitoring, and curriculum-based measurement (CBM). 
Although these types of assessments do not have agreed-on definitions, they overlap somewhat in practice. 

Formative assessment, progress monitoring, and CBM each are typically described as part of an ongoing process in which 
classroom teachers assess students' knowledge and understanding with activity-embedded, brief, small-scale tasks that are 
linked directly to the current curriculum topic. Formative assessments are not always standardized across schools, class- 
rooms, or even students; therefore, aggregating formative assessment data is not typically done or useful (Perie, M arion, 
Gong, & Wurtzel, 2007). Several studies on formative assessment suggest that teachers can use classroom-embedded stu- 
dent assessments to elicit achievement gains (Black, H arrison, Lee, M arshall, & Wiliam, 2002; Brookhart, 2001; Christman 
et al ., 2009; Hayward, Priestley, & Young, 2004; Heritage, 2007; Shepard, 2005). Based on a review of the literature on 
formative assessments, Black and Wiliam (1998a) concluded that formative assessments can increase student achieve- 
ment. In a related piece, Black and Wiliam (1998b) suggested that formative assessments are effective because — by defini- 
tion— they use evidence to directly inform teaching practices to meet students' learning needs, unlike summativeand other 
assessments. Drawing from eight studies on the impact of formative assessment on quantitative comparisons of learning 
gains through quasi -experimental and experimental designs, the authors concluded that increases in formative assessment 
practices lead to learning gains. None of the eight studies showed that increases in formative assessment practices nega- 
tively affected student achievement. However, the magnitude of the increases in student learning varied according to how 
teachers responded to the information that formative assessments provided and how they used the assessment information 
to provide feedback to students about their progress. Based on this finding, Black and Wiliam (1998b) called for further 
research, specifically on how teachers can best use assessment feedback to improve student learning. 


1 The America Creating Opportunities to M eaningfully Promote Excellence in Technology, Education, and Science (COM PETES) Act (20 U .S.C. 9871) aims to improve 
American competitiveness in science, technology, engineering, and mathematics. Among its provisions are requirements for a statewide PK - 16 education data system 
(see section 6401(e)(2)(D)). 
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CBM is a type of student progress monitoring characterized by quick (one- to five-minute) student-level assessments that 
teachers administer weekly to measure student progress in an academic area. The CBM approach to student testing was 
developed as an alternative or supplement to commercially distributed standardized tests. In the special education literature, 
several studies on the effects of CBM on learning outcomes of students with disabilities provided early insight into data- 
driven teacher practices that positively affect student achievement (e.g., Deno, 1985; Deno & Fuchs, 1987; Fuchs, Deno, 
& Mirkin, 1984). CBM specifies procedures for measuring student proficiency within curricular goals and basic skills. 
Some key uses of CBM are screening students for special services, developing and monitoring instructional programs, and 
evaluating program efficacy (Fuchs & Fuchs, 1990). In a study by Fuchs et al. (1984), students with disabilities were ran- 
domly assigned to either CBM treatment or a more traditional special education evaluation over a period of 18 weeks. The 
researchers observed teacher pedagogy and concluded that teachers assigned to use CBM were more responsive to their stu- 
dents' needs and achievements than those in more traditional special education settings. M oreover, their students achieved 
greater outcome gains than their peers in the control group. Similarly, Fuchs and Fuchs (1990) also demonstrated that CBM 
increased student learning outcomes, and CBM with additional support (in the form of skills analysis) increased student 
performance more than using CBM alone. This result emphasizes the focus of CBM on providing teachers with instructional 
guidance, in addition to providing high-quality tools for ongoing assessment, to impact student learning and understanding. 

Another study of classrooms in 24 states found that student achievement increased in classrooms implementing curriculum- 
based progress monitoring and instructional management systems as part of a mathematics program, as compared to control 
classrooms, with significantly higher achievement for classrooms where these practices were implemented with high fidel- 
ity (Ysseldyke & Tardew, 2007). Other studies of CBM report similar findings, demonstrating that CBM increases student 
achievement (Davis & Fuchs, 1995; Hintze, Daly, & Shapiro, 1998; Marston et al., 2007; Stecker & Fuchs, 2000), and 
perhaps more important, when coupled with instruction in effective teaching strategies, it increases student outcomes more 
than CBM alone (Fuchs, Fuchs, & Hamlett, 2007; Fuchs, Fuchs, Hamlett, & Stecker, 1991). 

The existing evidence about formative assessments in general and their use in specific contexts (e.g., special education) pro- 
vides a basis for using regular, systematic assessment to inform instruction. However, research has yet to clarify whether the 
widespread use of assessments— commonly known as interim assessments— can produce robust gains in student achieve- 
ment. 

Of particular interest in this study are interim (also known as benchmark) assessments that are typically adopted at the dis- 
trict level. Interim assessments are generally defined as assessments that are administered at regular intervals throughout 
the school year to help educators gauge student achievement before the annual state exams used to measure adequate yearly 
progress (AY P; Christman et al., 2009). Interim assessments provide data that can be aggregated to the student, teacher, 
and school levels and are often designed to predict student performance on end-of-year accountability assessments. Other 
purposes of interim assessments are to provide information to diagnose student strengths and weaknesses and to provide 
evaluative information about curricula or instructional programs (Perie et al., 2007). Interim assessments are administered 
routinely (e.g., every six to eight weeks) across grade levels in particular content areas (e.g., reading and mathematics) 
within a school or a school district. They may be commercially developed, developed by school districts or states, or a 
combination of both. Some interim assessments are delivered as fixed form tests, whereas others are delivered as computer- 

adaptive tests based on large item banks. 
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Not surprisingly, some evidence suggests that such assessments are not sufficient by themselves to raise student achieve- 
ment. In a large study of the effectiveness of interim assessments, Henderson, Petrosino, Guckenburg, and Hamilton (2007; 
2008) reported that assessments used as part of a pilot program in M assachusetts did not yield improvements in student 
mathematics achievement. In the study, 22 schools used state-developed quarterly administered assessments. Using aquasi- 
experi mental design, the researchers matched these schools in terms of student population to 44 similar schools. They com- 
pared the mathematics achievement scores of eighth-grade students at schools with state quarterly assessments with those of 
their counterparts in schools without state quarterly assessments. The authors did not observe any statistically significant or 
substantively important differences between the two groups. However, they noted that other interim assessments may have 
been in place at the comparison schools, and, perhaps most important, information about how the data from the assessments 
were used by educators in the treatment schools was beyond the scope of the study (Henderson et al ., 2007). 

Indeed, other research suggests that the effects of interim assessments on improving student achievement depend on the 
ways that interim assessments are implemented and used (M arshall, 2006). The use of interim assessment data is the direct 
focus of the key dimensions of data use and the theory of action that we posit in this study. 

Key Dimensions of Data Use 

To guide the study, we formulated an integrated theory that hypothesized how data practices at multiple levels (school dis- 
trict, school, principal, teacher, and student) may be related to student achievement. 

We began by acknowledging interim assessment data can be used for three general implicit purposes: 

• To better understand the academic needs of individual students and respond to these needs by 
targeting instruction, support, and resources accordingly. 

• To better understand the instructional strengths and weaknesses of individual teachers and use 
this information to focus professional development, peer support, and improvement efforts. 

• To support and facilitate conversations among teachers and instructional leaders regarding 
strategies for improving instruction. 

These practices, in turn, are thought to lead to improved and more responsive teaching and, therefore, yield increased stu- 
dent achievement. 

From this broad theoretical perspective, our goal was to articulate a theory of action that would undergird our investigation 
of the relationships among data-use practices and improvements in student achievement over time in large urban districts. 
Our intention was to ground the specific classroom- and school-level data-use practices that could theoretically improve 
student achievement in the context of the larger systems in which they occur. 

At the outset, we identified four key dimensions of the interim assessment data-use process. In the following sections, we 
describe the theory of action in terms of these four dimensions. 

1. Contact State, school district, and school conditions that facilitate a general emphasis on using data (i.e., 
a "culture of data use") 
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2. Supports for cbta use Concrete factors that facilitate, enable, and support specific uses of data 

3. Wrking with data The manner in which teachers and principals work both individually and col labora- 
tively to review data and identify specific ways the data are used to improve knowledge about student 
needs 

4. Instructional res p onses The ways that teachers can respond to the knowledge and information generated 
by their review of student data, which in theory could lead to improved teaching and learning and higher 
student achievement outcomes 

This proposed theory of action is shown in Exhibit 1.1. The broader dimensions and perceptions are on the left side of the 
diagram, and more specific aspects of data-use practices are on the right, with the ultimate end goal of improved student 
achievement on the far right. The diagram flows from left to right, with key dimensions on the left leading, in theory, to key 
dimensions to the right. 


Exhibit 1.1. Using Data from Interim Assessments to Improve 
Student Achievement 
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In brief, the Conteixt factors at the far left are hypothesized to support the establishment of Supports for Data Use which 
in turn are expected to facilitate teachers and principals Working with Data. Waking with Data hypothetically leads to a 
change in teaching strategies or Instructional Responses, which ultimately leads to improved student achievement. The 
theory of action also explicitly acknowledges that real or perceived Barriers to Data Use can affect the key dimensions and 
the links among them. Barriers a re represented in Exhibit 1.1 below the key dimensions, with dashed arrows signifying that 
real or perceived barriers can interrupt the theoretical process of supporting and using data to improve student achievement 
at any point. 

Along with the left-to-right flow of the key dimensions, we also posited that the ways educators work with data must first 
result in a change in their knowledge and then lead to a change in instructional responses. That is, to make an instructional 
change in the classroom with all or some students, a teacher requires an improved understanding of what students do or do 
not know or understand. However, we did not define teacher knowledge as a key dimension of practice because our focus is 
on observable, measurable aspects of data use. Nevertheless, in the theory of action, we identify improved knowledge as a 
mediating step between Working with Data and Instructional Responses 

Although the arrows in the figure illustrate some of the key relationships, Exhibit 1 . 1 does not illustrate every important 
relationship. For example, the elements of practice listed within each key dimension may theoretically contribute to and 
reinforce one another. For example, although staffing resources and organizational supports both contribute to individual 
teacher attention to data, staffing resources and organizational supports also may reinforce one another. Moreover, although 
the theory of action implies that each key dimension is primarily related to the next key dimension to the right, any of the key 
dimensions might theoretically be linked with student achievement directly orthrough other unmeasured mediating factors. 

In the following sections, we review the literature that informs the theory of action, focusing on the four key dimensions. 
Under each dimension, we define the key elements within the theory of action in Exhibit 1.1. Following a review of the 
relevant literature is a list of important components of each key element that provide a framework for the broader study. 
Finally, we review literature on typically cited barriers to data use— that is, factors that are known or hypothesized to limit 
the extent to which teachers use data to inform their instruction. 

Key Dimension 1: Context 

The first key dimension of data use broadly encompasses the contextual and cultural factors that may be related to data use. 
K ey elements of Context include the assessment context, the instructional context, the state and district data culture, and the 
school data culture (see Table 1.1). Although other contextual elements are theoretically relevant (e.g., the political or the 
economic context), our theory and measurement of data use focus on factors that we hypothesized are most relevant to the 
use of data in school districts, schools, and classrooms. 
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Table 1.1. Summary of Aspects of Context 


Key Elements 

C omponents 

Description/E xamples of C omponents 

Assessment 

Context 

Purpose of assessment 

T o improve instruction and predict performance on 
accountability tests 


Assessment type 

Curriculum embedded; external benchmarks; formative 
and summative assessments; test construction (e.g., 
externally versus internally developed); length and 
structure; fixed forms versus item banks; cumulative 
versus unit based 


Assessment quality 

Reliability and validity; appropriate scaling; alignment 
with state standards, state assessments, curricula, and 
pacing guides 

1 nstructional 
Context 

Implementation of 
districtwide curricula and 
pacing guides 

Flexibility of curricula; flexibility and speed of pacing 
(including time to reteach) 


Centralized versus site- 
based decision making 

Decisions related to curriculum and instruction, 
staffing/human resources, and professional 
development 


Accountability context 

Includes district and school AYP history 


Schoolwide and/or 
districtwide initiatives 

Implementation of other districtwide or statewide 
instructional initiatives; implementation of systemwide 
or schoolwide response to intervention (RTI) strategies 

State and 
District Data 

Support 

Explicit support for the use of data (i.e., as an explicitly 
stated state or school district priority) 

C ulture 

Clearly articulated goals 
and expectations 

Clearly articulated plan for implementing process and 
procedures to support and encourage data use; clear 
goals for using data across the system 


State and district 
participation 

State- and school district- level perceptions of validity, 
relevance, and quality of assessments; participation in 
discussions about data integration into state and district 
reviews, evaluations, and goal setting 

School Data 
C ulture 

Clearly articulated goals 
and expectations 

Presence and staff awareness of an action plan or clear 
strategy for data use; level of buy-in for data strategy; 
shared goals and norms around using data and interim 
assessments; clearly articulated process for 
implementing the data system 


Stakeholder knowledge and 
perceptions 

Quality, usefulness, and usability of data and reports; 
usefulness of training and coaching activities; 
accountability for data use (e.g., using data is part of 
job expectations) 


Support and guidance from 
school- and school district- 
level leadership 

Participation in planned meetings to discuss data; 
interest in and availability for additional conversations 
about data 
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Assessment Context. Assessment context includes goals, expectations, and policies related to developing and implement- 
i ng i nteri m assessments, i nd udi ng the types of assessments given and thei r purpose(s) . A ssessment context al so i ncl udes the 
quality (e.g., validity and reliability) of the interim assessments. 

Research suggests that the perceived quality of the data is as important as the strength of the data infrastructure and the ac- 
cessibility and timeliness of the data. The perception that assessments are of poor quality can be a clear barrier to use. Stud- 
ies have concluded that doubts about the accuracy of data lead to a lack of support for data initiatives and result in decreased 
data use by teachers (e.g., Feldman & Tung, 2001; Herman & Gribbons, 2001; Herman, Yamashiro, Lefkowitz, & Trusela, 
2008; Ingram, Louis, & Schroeder, 2004). One evaluation of districtwide use of data found that concerns about the accuracy 
of student data were correlated with lower levels of data use(Wayman, Cho, & Johnston, 2007). Similarly, Kerr, M arsh, Ike- 
moto, Darilek, and Barney (2006) found that the perceived validity of data affected the extent to which teachers used them. 

The alignment between the selected assessment and the intended uses of the data is also important. M ilitello, Schweid, 
and Sireci (2010) emphasized the importance of a close fit between the characteristics of a formative assessment and the 
intended use of the data by the school district and teachers to ensure that the data have a meaningful impact on teachers and 
students. 

Indeed, multiple studies have found that the alignment of the interim assessments with standards, state tests, and pacing 
guides facilitates data use(Kerretal., 2006; M arshall, 2008; U.S. Department of Education, 2010a). For example, in Phila- 
delphia, researchers found that interim assessment results were viewed as highly relevant to teachers' instructional planning 
because they were aligned to the curriculum and a six-week instructional cycle. The sixth week of the cycle was designated 
for remediation and extension of topics that could be designed by teachers on the basis of their review of the interim assess- 
ment results (Christman et al., 2009). However, in the same study, teachers' satisfaction with the interim assessments was 
not significantly related to student achievement growth in reading or mathematics. 

Instructional Context. The instructional context includes the curricular and instructional environment in which teachers 
and principals collect and use data. The uniformity, focus, and history of the instructional program all have the potential to 
affect how data are used in school districts, schools, and classrooms. As is the case with assessment context, instructional 
context is primarily meant to capture issues at state and district levels that shape school and classroom data activities. 

A key aspect of instructional context is the degree of flexibility in the curriculum and pacing schedule. The literature sug- 
gests that school districts and schools must be flexible in their curriculum pacing to allow teachers time to alter instruction 
on the basis of assessment results (Cl une& White, 2008; Datnow, Park & Kennedy, 2008; David, 2008). M arsh et al . (2006) 
suggested that curriculum pacing pressures— especially in the presence of regimented programs with pacing plans— are an 
obstacle to teacher data use. Even if pacing pressures are more perceived than real, teachers often follow the pacing plans 
instead of adjusting their instruction on the basis of the results of their data analyses (M arsh et al., 2006). 

State and District Data Culture. State and district data culture includes attitudes, direction, and support at state and district 
levels regarding using data in general and interim assessments in particular. We hypothesized that the degree and nature of 
support for data use, as well as the direction of district and state policy in this area, can affect the manner and extent to which 
data are employed at the school level. 

M arsh etal. (2006) found that teachers use data more frequently in school systems whose principals had committed to data- 
driven decision making and had a clear vision about data use at the school level. These school systems also were character- 
ized by openness and a sense of collaboration around data use, in contrast to school systems in which data analysis was seen 
as an individual activity. 
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Clearly articulated and communicated goals for district data use also appear to be related to the extent to which teachers 
use data. Studies have found that a barrier to teacher data use is the perception— real or imagined— that teachers are going 
to be blamed for the poor performance of their students (Clune & White, 2008; Ingram et al 2004; Kerr et al 2006; M ar- 
shall, 2008; Park & Datnow, 2009). Conversely, Park and Datnow (2009) found that school districts can support data use in 
several ways, including aligning resources and goals, providing professional development, modeling effective data use, and 
encouraging shared decision making and collaboration. A correlational study by Anderson, Leithwood, and Strauss (2010) 
reinforced the role of district leadership in setting expectations for data use. They found that district leadership was more 
strongly related to some of the typical barriers to data use, such as accessibility, timeliness, quality, and capacity for use, 
than was leadership among school principals. 

School Data Culture. This key element of Contextconcerns goals, norms, expectations, processes, attitudes, and leadership 
for using interim assessment data at the school level. 

One study of Philadelphia's interim assessments found that the key supporting factors that facilitated the link between using 
the interim assessments and academic progress were school-level instructional leadership and collective responsibility for 
data (Christman et al., 2009). This was true in a context where although teachers expressed satisfaction with the benchmarks 
and their alignment with the core curriculum, the pacing plan, and the instructional cycle, their satisfaction alone was not 
predictive of growth in student achievement in reading and mathematics (Christman et al., 2009). 

Other research also suggests that school leadership is a key factor in the successful use of data. Kerr et al. (2006) found 
greater data use in schools that had created data-driven cultures through strong school and district leadership. M urnane, 
Sharkey, and Boudett (2005) found that teachers' own use of data depends largely on the amount of principal support for 
data use. Many other studies also highlight this point (e.g., Enright & Witham, 2008; Feldman & Tung, 2001; Lachat& 
Smith, 2005; M arsh et al., 2006; M ason, 2002). Anderson etal. (2010), however, found that although principals play a key 
role in influencing data use in their schools, the majority often do not act to change the specific conditions that affect data 
use that are under thei r control . 

Although the principal often is identified as the leader responsible for several important data supports, case studies by the 
U.S. Department of Education (2009) suggest that other individuals, including coaches and lead teachers, may also provide 
important leadership support for data use. Indeed, schools with higher levels of data use often had more widespread data 
expertise than typical schools because they did not confine the expertise to the principal or a lead teacher (Anderson et al., 
2010 ). 

Key Dimension 2: Supports for Data Use 

Supports for Data Use involve the specific elements of practice related to logistical and operational support for using data, 
including the infrastructure, organizational resources, time allocation, and personnel resources necessary to support using 
interim assessment data to guide and improve instruction. This dimension is related to the amount of investment and sup- 
port that exist at the district level but is focused on the tools and resources that are available at the school level. It includes 
both technology-related resources and the content of the data and reporting system itself. In particular, the key elements in 
Supports for Data Use are data infrastructure, organizational supports, and staffing and human resources (see Table 1.2). 

The concrete supports that school districts and schools can provide to enhance data use appear to be important factors in 
whether data are used. In an article summarizing their two studies of interim assessment data use in Philadelphia, Bulkley, 
Christman, Goertz, and Lawrence (2010) assert that interim assessments can serve an instructional purpose, but critical to 
such use are the supports provided by the school district, including data systems, useful reports, time for reflection and col- 
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laboration, and professional development. Marsh et al. (2006), reflecting on conclusions drawn from four studies conducted 
by RAND, also suggested that concrete support for data use is critical to encouraging teachers to use data. The concrete 
supports they emphasize include various infrastructure supports, such as data access, the timeliness of the data reports, and 
adequate time for teachers and principals to review and discuss data. 

Table 1.2. Summary of Aspects of Supportsfor Data Use 


Key Elements 

C omponents 

Description/E xamples of C omponents 

Data 

1 nfrastructure 

Data access and 
dissemination 

T ype of access: availability of direct access into the data 
system and provision of district-generated reports; level and 
ease of access by all stakeholders (e.g., school district, school, 
principal, teacher, student, and parent); availability of 
computer resources; ability to manipulate data (e.g., 
disaggregation by item types or student subgroups); frequency 
and timeline of reports 


Content and 
capacity of 
reports and data 
systems 

Identification of student needs and classroom- and school- 
level needs and challenges; data disaggregation (i.e., student 
performance by subgroup, content standards, and item types) 

Organizational 

Supports 

Allocation of 
time 

Time allocated to access, review, and/or discuss data; time 
allocated for one-on-one meetings between teacher and 
principal or data leader to review and discuss assessment data 


Administrative 

support 

Principal participation in data-focused meetings 


M onitoring and 
implementation 
support 

District- and school-level oversight; evaluation; reporting 
requirements; tools for reviewing and understanding data 

Staffing and 
Human Resources 

Personnel 

resources 

School staff with the role of supporting access and analysis of 
data; district-level staff available to support data use; the 
functions and goals of data support staff (e.g., instructional 
versus accountability versus progress monitoring); availability 
of data support staff (i.e., percentage of time dedicated to 
providing data use support) 


Staff capacity 

Prior training and expertise of staff (e.g., assessment literacy, 
awareness, and experience using available data systems) 


Professional 
development 
and training 

Professional development focused on how to use data 
effectively; availability and access to experienced data 
coaches; ongoing evaluation of the success and effectiveness 
of training activities; level of participation in training activities 


Data Infrastructure. This key element consists of two primary components: the infrastructure for accessing, analyzing, 
and disseminating data and the content or capacity of the reports and the data system. Several studies have emphasized the 
importance of system-level i nfrastructure support (Datnow etal., 2008; K err et al ., 2006; M urnaneetal., 2005; Wayman & 
Stringfield, 2006). A U.S. Department of Education (2009) study found that data systems are often of limited use to teachers 
for instruction because of limitations in the data, the user interface, or system tools. For example, only slightly more than 
one half of teachers with access to a data system also reported having access to thei r students' diagnostic test performance. 
H owever, 79 percent of school districts report having an assessment system that analyzes interim assessment data, suggest- 
ing that at least in some districts, there is a disconnect between the actual available infrastructure and teachers' knowledge 
or perceptions about the available infrastructure. 
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The quality of the data infrastructure also seems to affect levels of teacher data use. Kerr et al. (2006) found that schools 
demonstrating greater data use had better data infrastructure systems that included timely reporting of results, online access 
to data, and an interface that allowed teachers to manipulate data and run specialized reports. 

Other studies concur that the timeliness and accessibility of data are particularly important. For example, Clune and White 
(2008) concluded that out-of-date data significantly impede the ability of teachers to modify their instruction. Specifically, 
teachers and administrators reported that it would be preferable to receive data within two weeks after assessments were 
administered. Otherwise, the data were considered too out of date for teachers to modify instruction in the current quarter 
school year. Similarly, in a study of data-use practices and perceptions in five urban high schools, interviews with teachers 
and principals revealed that those who had access to timely data were more likely to use them and were more successful at 
integrating results from their analyses into classroom practice (L achat & Smith, 2005). 

Organizational Supports This key element refers to logistical and operational supports for data use, including scheduling 
and allocating time for review and discussion of interim assessment data and their implications for instruction. A I though the 
presence or prevalence of these supports may be a function of the data culture within a school or a school district, the focus 
in this dimension is on concrete systems that exist apart from norms, expectations, and other soft supports. 

A supporting factor for using data in schools is the allocation of ti me for teachers to work independently and col laborati vely 
with student data. Inversely, the lack of structured time to leam how to use and review data and reflect on instructional re- 
sponses is often cited as a barrier to effective data use. A report by the U .S. Department of Education (2009) noted that the 
majority of teachers who use student data report doing so on their own and with colleagues in their departments or grade 
levels— in grade-level team meetings that are sometimes facilitated by a coach. 

A study by Young (2006) also found that data use is more likely to occur when schools and school districts provide struc- 
tured time to allow teachers to learn how to use data collaborati vely. The author used an embedded-systems perspective to 
explore factors influencing teachers’ data use. Specifically, she explored how different factors from the hierarchical structure 
of school systems — such as grade-level team norms and leadership at the school and district levels — influence how teachers 
both view and use data. The findings were based on 90 interviews with district principals, school principals, and teachers, as 
well as 73 observations of grade-level team meetings, staff meetings, and professional development sessions. 

In addition, tools such as assessment results linked to model lesson plans, frameworks, and curriculum guides can be devel- 
oped within the data system to help teachers interpret data and respond instructional ly; however, these are not common- 
even in districts known to be "high data users" (U .S. Department of Education, 2009). 

Staffing and Human Resources. This key element refers to a school's capacity to use data to improve instruction. It in- 
cludes staff positions, the capacity of staff to use data, and professional development available to support data use. 

A lack of staff capacity and a lack of training in assessment and data analysis have been reported as important obstacles to 
teacher data use in numerous studies (Heritage, 2007; Heritage & Bailey, 2006; Herman et al., 2008; Ingram et al., 2004; 
L achat & Smith, 2005; Sharkey & M urnane, 2006; U.S. Department of Education, 2009; Wayman et al ., 2007). Herman et 
al. (2008) found that teachers are typically not trained in assessment and are not introduced to the content and pedagogical 
knowledge required to interpret student performance results and make instructional changes. 

Professional development in using data provided by instructional coaches or other data facilitators can increase the likeli- 
hood that teachers will use data. Several studies have suggested that trained teachers are more apt to modify their teaching 
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practices appropriately on the basis of the knowledge they have gained from assessment data (Henke, 2005; M arsh et al., 
2006; M ason, 2002). Young (2006) conducted observations and interviews with district principals, school principals, and 
teachers about data use and concluded that effective data use was more likely to occur when districts modeled data use for 
their teachers. 

Based on the 2006-07 National Educational Technology Trends Study (NETTS) teacher survey, 39 percent of teachers self- 
reported that the training they received about data-driven decision making prepared them to use data to improve student 
achievement (U.S. Department of Education, 2009). In addition, the NETTS study also found that school districts with high 
levels of data-driven decision making tend to offer district-funded, school-based data coaches to support teachers' data use. 
The role of the data coach varies, but the typical responsibilities include helping teachers examine and interpret data and 
connect results to instructional strategies (U .S. Department of Education, 2009). 

Key Dimension 3: Working with Data 

Working with Data includes the specific ways in which classroom-, school-, and district-level staff review and understand 
interim assessment data, interact with one another regarding assessment data, and use these data to inform their knowledge 
of student needs and decision making regarding instructional strategies. I n other words, what do they do with the data they 
receive? How do they interpret and use the data to inform what they know about student needs? The key elements of Work- 
ing with Data are related to the ways that teachers and principals work individually and together to understand student data, 

including individual teacher attention to data, collaboration around data, and “making sense of data,” which refers to specific 
ways of reviewing assessment data to understand student performance (seeTable 1.3). 
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Table 1.3. Summary of Aspects of Wbridng with Data 


Key Elements 

C omponents 

Description/E xamples of C omponents 

Individual 

Teacher 

Frequency of 
data review 

Interim assessment data and/or reports 

Attention to 
Data 

Focus and 
purpose of 
review 

U nderstanding the data or reporting system; using data to identify 
and discuss student-, classroom-, or school-level patterns or needs; 
reflecting on instructional challenges; identifying potential areas to 
incorporate into lesson plans 

Collaboration 
Around Data 


F requency of teacher meetings about student data; teachers 
partnering outside of formal meetings ( e. g . , one-on-one peer 
support for interpreting data and developing instructional 
responses) 


T eacher 
collaboration 

Level of teacher, principal, and other staff (e.g., district liaisons) 
participation in meetings about data or involving data discussions 



U nderstanding data systems and interpreting assessment results; 
using data to identify and discuss student-level issues; discussing 
classroom- and school-level patterns in data; using data to reflect 
on instructional challenges and potential solutions 


T eacher- 
principal 
collaboration 

Recognizing accountability versus support; using data to identify 
and discuss student-level issues; discussing classroom- and school- 
level patterns in data; using data to reflect on instructional 
challenges and potential solutions; informal interactions about 
interim assessment data, student results, and student needs 


T eacher-coach 
collaboration 

Frequency and focus of meetings and discussions about data; using 
data to identify and discuss student-level issues; identifying 
classroom- and school-level patterns; using data to reflect on 
instructional challenges and potential solutions; working on 
instructional solutions (e.g., lesson planning and modeling) 


T eacher-student 
collaboration 

Providing explicit instruction to students on how to use their own 
achievement to monitor their progress; motivating students by 
setting clear, attainable goals; providing students with a clear 
understanding of thecontent and skills that will be assessed; 
providing time for reflection on students’ own performance 

Making Sense 
of Data 

Identifying 
students for 
additional 
support 

Comparing individual student scores with the performance of a 
larger group (e.g., class or grade level); identifying “bubble” 
students (students below but close to proficiency); identifying 
(diagnosing) students with particular needs in foundational skills 
(e.g., literacy); identifying students for intervention within the 
classroom; targeting students for intervention outside the classroom 
(supplemental or pullout) 


Identifying 
specific student 
needs 

Identifying students with particular needs in specific concepts; 
grouping students on the basis of similar patterns or trends over 
time; reviewing scores on individual items to understand patterns of 
performance and to diagnose areas of misunderstanding (e.g., item 
analysis); reviewing individual scores by content standards and 
item types; reviewing student growth over time 


Identifying 
school- and 
classroom- level 

Reviewing average scores to determine class strengths and 
weaknesses; reviewing classroom-level scores by content standards, 


instructional 

issues 

item types, subgroups; reviewing student growth over time 
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We hypothesized that these elements of practice are mutually reinforcing. For example, teacher collaboration around data 
elicits individual teacher attention to data and vice versa. Specific ways to make sense of data are facilitated by the time 
spent in collaboration around data (and by individual teacher attention to data), and they are a function of theextentto which 
collaboration around and individual attention to data emphasize these particular activities. Collaboration around data, indi- 
vidual teacher attention to data, and specific ways to make sense of data can lead to discussion and review of instructional 
strategies and responses. This entire process leads to improved teacher knowledge of student needs and available instruc- 
tional responses. This in turn leads to specific instructional responses (our fourth category of practice). 

IndMdualTeacherAttaitiontoData Research suggests that simply having interim assessments in place is not enough- 
knowing how to use data to inform instructional practice is necessary to improving student achievement. Specifically, teach- 
ers can use data to identify problems, then to identify reasons behind the problems and further, to determine how to take 
appropriate actions to solve them (Anderson etal., 2010). 

Easy-to-access data-management systems can allow users to analyze data to determine problem areas, but levels of actual 
usage appear to vary both within and across school districts. In an in-depth study of data system usage, Tyler (2010) found 
limited use among teachers in the Cincinnati Public Schools. On average in 2008-09, teachers of mathematics, English, 
social studies, and science in Grades 3-8 spent about seven hours using the system over the course of the year, with the 
majority of their time being devoted to viewing pages that provide resources for teaching (32 percent of time) and pages that 
have results for multi pie students in thei r class (27 percent of time). A I though higher percentages of teachers logged into the 
system in the weeks foil owing an interim assessment (from 45 percent to about 70 percent), an average over the year showed 
that only from 10 percent to 40 percent of teachers logged in during any given week and spent an average of six to eight 
minutes using the system. Focus groups provided some possible insights into why usage was so low, including a perceived 
lack of validity of the interim assessment, a lack of time to reteach what the data system identified as weaknesses, and a lack 
of time provided to use the data system. 

Making Senseof Data When teachers do review student assessment data, multiple research studies have noted some com- 
monly cited uses, including identifying individual or groups of students with particular needs (H enke, 2005; Love, 2004; 
Niemi, Vallone, Wang, & Griffin, 2007), identifying bubble or at-risk students whose scores fall within particular ranges 
(Blanc et al., 2010; Christman et al„ 2009; Long, Rivas, Light, & M andinach, 2008; M arsh et al ., 2006), and comparing 
classroom scores with school scores (Niemi etal., 2007). 

A study in Philadelphia's public schools revealed that teachers consider a variety of factors when reviewing data (Nabors 
Olah, Lawrence, & Riggan, 2010). M ost teachers in the study began their review by identifying their classes' weak points. 
They considered their students' results in light of what they knew about their students' background or performance, their 
district’s curriculum or pacing guide, and their perceptions about the difficulty of the material for their students. They also 
used a triage method of focusing efforts on the students or topics deemed most in need of attention. 

H enke's (2005) case study of three school districts provides examples of how teachers and principals used data from district- 
level interim assessments in efforts to improve student achievement outcomes. Seven of eight schools in the Lemon Grove 
School District were ranked underachieving by the California Academic Performance Index. After implementing a new 
districtwide data system, three of the four schools that received Title I aid were declared high-achieving Title I schools. The 
author noted that implementation of the data system alone did not increase student achievement. Instead, improvement was 
attributed to the principals’ and teachers’ specific and targeted use of data to guide instruction and intervention. 
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For example, in one school, the principal analyzed state and district assessment results to identify students with particular 
weaknesses. The principal found that high percentages of fifth-grade students were performing below proficiency in vocabu- 
lary and reading comprehension. This knowledge shaped the school's literacy intervention, which in turn was believed to 
have led to improvements in student reading achievement. 

I n other cases, however, similar uses of data did not produce i ncreases in student outcomes, or, at best, student progress was 
unclear. For example, Quint, Sepanik, and Smith (2008) investigated the effects of the Formative Assessments of Student 
Thinking in Reading (FAST-R) program in the Boston Public Schools (BPS) using comparative interrupted time series 
analyses that compared student achievement outcomes before and after FAST-R was implemented. A preintervention base- 
line was constructed using data from the 2001-02 school year through the 2004-05 school year. Outcomes were predicted 
for the 2005-06 school year using the preintervention baseline, and the differences between actual outcomes and predicted 
outcomes were measured. Specifically, they examined differences in trends in third- and fourth-grade reading scores on the 
M assachusetts Comprehensive Assessment System in 21 BPS elementary schools and found that the FAST-R data-use pro- 
gram did not have a statistically significant effect on student achievement. However, the study concluded that the FAST-R 
program, and, in particular its data coaches, seemed to help teachers better understand how their students were performing, 
including identifying students who were weaker in certain areas of reading. 

Similarly, Flerman et al. (2008) explored the relationship between student achievement in 13 Seattle schools and the use 
of the Seattle Public Schools' Comprehensive Value-Added Assessment System using data from four case studies, district 
surveys, and student test scores. The results of their latent variable, multilevel analysis suggested that data use did not have a 
statistically significant effect on student outcomes. The extent to which the lack of impact on student achievement stemmed 
from variations in data practice, a small sample size, or the lack of effective and widespread data use is unclear, but it ap- 
peared that the teachers valued access to student achievement data. The case studies suggest, however, that teachers do use 
data to inform their teaching practices and improve student outcomes. 

Collaboration around Data Shown in the theory of action as collaboration among teachers, between teachers and prin- 
cipals, coaches, and students, collaboration around data includes the supports and practices related to time that teachers 
and principals spend examining student data in collaboration with others. Collaboration around data occurs when teachers 
discuss student achievement data with other teachers, principals, coaches, parents, and students themselves. Herman et al. 
(2008) found that in at least one of the four case-study schools, the principal reported that most teachers collaborated fre- 
quently to compare student data across classrooms and grade levels to better prepare students for the next grade. A nother 
recent study found that principals were more often facilitating teachers' data use than using the data independently, thus 
supporting collaboration around data within their schools (Anderson et al., 2010). The 2007 National Educational Technol- 
ogy Trends Study also found that teacher collaboration around data was almost as common as teachers' individual use of 
data systems (U.S. Department of Education, 2009). M arsh, M cCombs, and M artorell (2010) found a small but statistically 
significant link between student achievement and the frequency of meetings between coaches and reading teachers to review 
assessment data. 
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Involving students in the review of their own data also has been noted as an important aspect of effective data use. Work 
conducted in the 1990s documented that interventions that incorporated students analyzing their own data combined with 
feedback from their teachers seemed to improve student outcomes (Phillips, Hamlett, Fuchs, & Fuchs, 1993). Another more 
recent study that used random assignment found that granting students access to an online-based feedback system that in- 
cluded students' individual test scores led to increased student achievement (M ay & Robinson, 2007). According to recom- 
mendations from a recentU.S. Department of Education Practice Guide titled Using Student Achievement Data to Support 
Instructional Decision Making, students should be active partners in analyzing their own achievement data. The authors 
specifically suggest that it is important to explain expectations and assessments to students, including the content and skills 
that will be assessed (Hamilton et al ., 2009). Also, similar to facilitating factors of data use at the teacher and school levels, 
feedback to students should be timely, clear, and constructive (Black & Wiliam, 1998a; Brunner et al., 2005). 

Improved Knowledge 

Implicit in the path from making sense of data to implementing instructional responses is a change in educators' knowledge. 
That is, if data use is to be an effective means for improving instruction, it is likely that it must first yield improved teacher 
knowledge about student needs and principal and district knowledge about teacher and school needs. 

Improved Teacher Knowledge involves improved awareness and understanding of the following: 

• Instructional needs and challenges of individual students 

• Instructional needs and challenges facing their classrooms as a whole 

• Teachers’ own strengths and weaknesses 

• Strategies and resources for addressing the needs of struggling students 

• Different strategies for teaching and reteaching specific concepts 

Improved Principal and District Knowledge involves improved awareness and understanding of the following: 

• Instructional needs and challenges facing individual classrooms or teachers and the school 
as a whole 

• Teachers’ (and schools’) strengths and weaknesses 

• Strategies and resources for addressing the needs of teachers and schools 

Although we identify teacher and principal knowledge explicitly in the theory of action, measuring teacher or principal 
knowledge (and changes in what they know as a result of reviewing data) was beyond the scope of this study. 
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Key Dimension 4: Instructional Responses 

Instructional Responses are the ways classroom-, school-, and district- level staff translate the improved knowledge they 
glean from reviewing interim assessment results and use it to change classroom-level instruction. This dimension also in- 
cludes instructional responses (e.g., interventions) implemented at the school and district levels in response to patterns and 
trends in student assessment data (Table 1.4). 

There is evidence that beyond access to data, knowing how to use data to inform instructional practice is related to improv- 
ing student achievement. Kerr et al. (2006), in a study of data use in three urban districts, noted that there is anecdotal evi- 
dence that data-based decision making can have a positive impact on student achievement. Given that a number of studies 
have shown that just administering the tests does not appear to yield changes in student achievement, we hypothesize that 
this process must move beyond using data to diagnosing problems to identifying appropriate actions to solve them (see also 
Anderson et al., 2010). 

Table 1.4. Summary of Important Aspects of Instructional Responses to Data 


Key Elements 

C omponents 

Description/E xamples of C omponents 

T eacher 1 nstructional 
Responses 

Establish/adjust 

groupings 

Establishing and/or adjusting student groups on the 
basis of assessment scores 


Change scope and 
sequence 

Changing the scope and sequence of individual 
lesson plans or larger curriculum maps 


Adjust lesson plans 

Adjusting the approach and materials used in lesson 
plans 


Review and reteach 

Covering material again for all or some students in 
the classroom, reteaching content that students may 
have missed during the initial instruction period 


Provide supplemental 
resources to targeted 
students 

Providing supplemental interventions and support 
(e.g., tutoring) for struggling students; additional 
attention to “bubble” students ( i . e. , students close to 
reaching proficiency) 

School Instructional 
Responses 

School-level 
instructional response 

Providing school wide responses such as 
professional development for teachers or revising 
the school improvement plan. 


Table 1.4 lists a number of instructional responses or changes that teachers and schools make in response to the review 
of student assessment data. Previous studies document that teachers respond in varied ways to student achievement data, 
including the components listed in Table 1.4 as Teacher Instructional Responses. For example, student data are commonly 
used to identify certain students for small-group instruction and interventions within and outside the classroom (Blanc et 
al., 2010; U .S. Department of Education, 2010a). In their study of 10 Philadelphia schools, Blanc et al. (2010) found that 
regrouping was one of the most common strategies that teachers used after viewing interim assessment results. Teachers 
identified weaknesses in groups of students and worked with them to address those challenges. Teachers described a flex- 
ible system, with groups that changed frequently based on data. I n an exploratory case study of three districts and a portion 
of a fourth district, Snipes et al. (2002) also found that data were used for grouping and providing interventions or tutorials 
within the classroom. 
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In an exploratory study using a mix of qualitative and quantitative analyses, Brunner et al. (2005) examined the use of 
assessments in New York City public schools in grades 3-8 and found that interim assessments were reported to be most 
useful for making pacing decisions and prioritizing instructional time. B runner etal. conducted several rounds of interviews 
and administered surveys in 17 schools across the city. Of the respondents, 89 percent said that assessments were useful in 
making prioritizing decisions regarding instructional time, and 76 percent said that assessments were important tools when 
planning lessons. A majority (51 percent) also reported that they use assessments in making year-long pacing plans. 

Clune and White (2008) examined how data from the quarterly assessments were used in the Providence Public School 
District. The assessment program was initiated in 2004, with interim assessments administered in grades 2-8 in language 
arts and mathematics. The results based on teacher surveys indicated that assessment data were useful in identifying low- 
performing students for remediation and adjusting the content of lesson plans to improve student learning. 

Another common instructional response for teachers is to reteach content to the entire cl ass or groups of students. For exam- 
ple, in the previously described study of Philadelphia's interim assessments, Nabors O I ah et al. (2010) examined the ways 
that teachers weighed their options for recovering particular content during the district's reteaching week. As noted under 
Key Dimension 3: Working with Data, the teachers used a triage method to identify the topics thatwere most problematic for 
students. Teachers in the study most commonly attributed student mistakes to procedural errors; consequently, their reteach- 
ing often focused on procedural steps. However, the Philadelphia assessment results appeared to be of little help in guiding 
teachers in correcting conceptual errors. Teachers often presented material with which students had struggled in a different 
way; however, the change was not related to an analysis of the assessment items but rather to a belief that being exposed to 
different methods of teaching was beneficial to students. Goertz, Nabors Olah, and Riggan (2009) also found that teachers 
used data to determine what to reteach and to whom, but only half of the teachers in the study changed how they taught the 
concept, and those changes were not always rooted in the data. 

Blanc et al. (2010) similarly found that one of the most common instructional changes based on data was reteaching the 
information identified on the interim assessments as a class weakness. In this study of 10 elementary schools in Philadelphia, 
including five detailed case-study analyses and interviews of district staff, the district and school staff reported that some 
teachers retaught the material using the same instructional strategies, but others felt it was important to change the teaching 
strategies to provide students with a different way to access the information. 

In another study of teachers' instructional responses to student achievement data, Hoover (2009) found that in a suburban 
Virginia district, the most common instructional responses were differentiating instruction to remediate and/or enhance 
learning (96.4 percent), reteaching topics or concepts (93.5 percent), and changing the pacing of future instruction (91.7 per- 
cent). Many teachers also remediated and retested for a specific unit (82.7 percent), regrouped students within the classroom 
(80.4 percent), or changed their instruction in some other way (42.9 percent). Sixty-four percent of the teachers reported that 
pacing prevented them from reteaching, but they incorporated strategies to address assessment results into their next units. 

The most commonly cited school-level uses of student achievement data include revising school improvement plans, revis- 
ing curricular choices (content as well as scope and sequence), and implementing or adapting schoolwide interventions. 
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For example, in a case study involving interviews with school and district personnel, Henke (2005) described a school dis- 
trict’s initiative to use data in targeted and specific ways to improve schools that were in underachieving status. One school 
conducted an analysis of state and district assessment results to identify students with particular weaknesses. School staff 
found that high percentages of fifth-grade students were performing below proficiency in vocabulary and reading compre- 
hension. This knowledge shaped the school's intervention, which involved implementing a mandated corrective-reading 
effort during the school day, a guided reading program after school, and the use of a three-hour support aid. The author at- 
tributed the increased academic success of students in this school to these interventions and data-based curriculum changes. 

A larger 2010 study from the U.S. Department of Education (2010a) examined 36 case-study schools and found that the 
most common uses of data at the school level were school improvement planning and curricular decisions. Setting class- 
room, school, and district goals also has emerged as a prevalent response to interim assessment data in several studies 
(Clune& White, 2008; M arsh et al ., 2006; M arshall, 2008; Niemi et al ., 2007; Young, 2006). 

Barriers to Data Use 

Barriers to Data Use associated with any of the aforementioned key dimensions may disrupt the process at any point in the 
theory of action, see Exhibit 1.1 (theory of action). That is, real or perceived barriers to data use can interrupt the theoretical 
progression toward improved student achievement. Several obstacles to effective data use have been described in the review 
of the key dimensions; this section summarizes often-cited barriers in previous research as follows: 

• Lack of time to engage in data exploration and reflection (U.S. Department of Education, 2010a) 

• Poor assessment quality or validity (Feldman & Tung, 2001; Herman & Gribbons, 2001; Herman et al., 
2008; I ngram et al ., 2004; K err et al ., 2006) 

• Lack of data accuracy (Wayman et al., 2007) 

• Lack of alignment with standards and pacing (Marsh et al., 2006; U.S. Department of Education, 

2009; U.S Department of Education, 2010a) 

• Lack of timeliness and accessibility of data (Clune & White, 2008; Lachat & Smith, 2005) 

• Limited staff capacity (Heritage, 2007; Heritage & Bailey, 2006; Herman et al., 2008; 

Ingram et al., 2004; Lachat& Smith, 2005; U.S. Department of Education, 2010a; 

Sharkey & M urnane, 2006; Wayman et al., 2007) 

• Negative perceptions of the use of interim assessment data for teacher evaluation 
(Clune & White, 2008; Ingram et al ., 2004; K err et al ., 2006; M arshall, 2008) 
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The following section describes the methods and procedures used to address the research questions posed. The general 
approach in the study was to link teacher and principal survey data with student achievement data to test the relationships 
between data-use practices and perceptions and student achievement. The investigation focused on both elementary grades 
(grades 4 and 5) and middle grades (grades 7 and 8). 

The overarching goal of the study was to understand the links between data-driven instruction and student achievement by 
examining two research questions: 

1. W hat are the relationships between teachers' data-use practices and perceptions and their students' 
achievement? 

2. What are the relationships between school polices, practices, and resources for data-driven instruction and 
student achievement? 

Samples 

This section describes the samples of districts, schools, principals, teachers, and students included in the analyses that ex- 
amine data use and student achievement. Overall sample sizes are shown in Table 2.1; these numbers reflect the number 
of individuals who were included in analyses of the links between data use and achievement in four analysis samples: (1) 
elementary grades mathematics, (2) elementary grades reading, (3) middle grades mathematics, and (4) middle grades read- 
ing. Schools were randomly selected within districts (described below) and invited to participate. The teachers included 
were those in participating schools who taught mathematics and/or reading in grades 4, 5, 7, or 8, and completed one or 
more of the data-use surveys. The students included were those who were in the surveyed teachers' classes in these grade 
levels during the 2009-10 school year. 

Table 2.1. Number of Districts, Schools, Principals, Teachers, and Students in the 

F our G roups of A nalysis Samples 


E lementary G rades M idd le G rades 

M athematics Reading M athematics Reading 


Districts 

4 

4 

4 

4 

Schools 

111 

110 

86 

85 

Principals 

102 

101 

76 

75 

T eachers 

593 

614 

471 

532 

Students 

14,354 

14,764 

38,583 

36,169 


The principal samples overlap across subjects completely because they were asked to respond to items about data use in both 
mathematics and reading in their schools. There is considerable overlap (92 percent) between the teachers in elementary 
grades mathematics and reading samples because most teachers taught both subjects. Teachers who taught both were asked 
to respond to items about data use in both subjects. Among middle grades teachers, only 7 percent taught both mathematics 
and reading; most taught one subject or the other. There is also overlap in the student samples across subjects because most 
students had both reading and mathematics scores. I n the next section, we describe the sample of districts, schools, princi- 
pals, teachers, and students including providing the total number of unique participants. ExhibitC.l in Appendix C provides 
a detailed depiction of the samplefrom the beginning of survey administration to the samples of schools, principals, teach- 
ers, and students used in the analyses. 
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District Sample 

The four districts that participated in this study were selected by drawing on data from a district- level survey that we ad- 
ministered to all 67 member districts of the Council of the Great City Schools in J une 2009. The survey asked about district 
interim assessments, data systems, and data use in member districts. One version of the survey was administered to the 
academic chief/curriculum coordinator and another to the research director of each district. The survey versions were simi- 
lar in their content and scope but modified to reflect each individual’s role within the district to provide a general overview 
of the state of current practice in using data to inform school- and classroom-level decision making across urban districts 
in the U .S. A total of 35 curriculum coordinators (52 percent response rate) and 54 research directors (81 percent response 
rate) completed the surveys. Between the curriculum coordinators and research directors, respondents represented a total of 
62 of the 67 districts in the Council's membership (94 percent). 

Based on the district survey results and additional supplemental information, we identified four districts that (1) had admin- 
istered interim assessments continuously for the past three years; (2) planned to continue administering interim assessments 
for at least the next several school years; (3) administered interim assessments at least three times in a school year; and (4) 
had a data system with the capacity to meet the requirements of a quantitative study that would link school- and classroom- 
level data-use practices with student achievement. The selected districts also had to be willing to participate in the in-depth 
study, which included a series of principal and teacher surveys and a two-day site visit. 

To set the context for the study, a brief description of each district as of 2010 follows a general overview of the types of data 
available in the participating districts via the state longitudinal data systems. The following sections briefly describe the 
participating districts and their data use climates. The study team collected information regarding the latter during site visits 
conducted in each district during the 2009-10 school year. During these two- to three-day visits, we collected information 
about the district's history and the background of the interim assessment process through focus groups and interviews. 2 

The four participating districts are each in a different state, with a different emphasis on data use. Three districts are in states 
with high data capacity— that is, they have in place all or nearly all the required elements for the America COM PETES Act, 
with plans to improveon them and implement the remaining elements (seeTableA.l in Appendix A). 

District 1 . As of 2010, District 1 served about 90,000 students in 126 schools with a staff of about 6,500 teachers. The 
district adopted interim assessments in 2003. The interim assessments initially covered all the content that students were 
expected to learn during the school year and were administered to determine to what extent students were making progress 
toward meeting state and district standards in reading and mathematics. However, the district transitioned to a new interim 
assessment model in 2008, whereby each assessment is intended to reflect the content that has been taught to students up 
to that time, according to the district curriculum and pacing guide. This transition started with high schools and later was 
adopted in middle and elementary schools. The mode of administration also has changed, from online administration to 
paper- based administration. 

These changes to the interim assessment occurred concurrently with other significant changes in the district. These included 
a transition from a site-based management system to a more centralized management model, particularly for curriculum, 
assessment, school schedules, and budgetary practices. Another change was the adoption of districtwide curriculum maps in 
2009. In 2009-10, District 1 administered interim assessments three times in most grades. 


2 M ore information about the data use and interim assessment- related practices in each district is provided in Using Data to Improve Instruction in the Great City Schools: 
Documenting Current Practices, which is available online ( www.cqcs.org ). 
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District 2 District 2 is one of the largest school districts in the country, serving approximately 311,000 students in 324 
schools, with a staff of about 14,800 teachers. The district is divided into four areas that vary geographically and demo- 
graphical ly. From the district's perspective, each area requires different levels and types of support regarding data use. In 
this district, schools maintain site-based decision-making autonomy; this autonomy carries over into the interim assessment 
process. 

In District 2, the development of the interim assessments stemmed from the need to determine whether students were meet- 
ing benchmarks defined by the district as well as the need for more immediate data regarding student progress during the 
school year. With the previous testing structure, the district could use only the prior year's state accountability test to plan 
for the upcoming school year. 

In this district, common assessments— generally developed by teachers with some guidance from district principals— also 
are used in some schools and administered approximately twice a year. The common assessments reportedly compete with 
the interim assessments, particularly in mathematics, because the teachers believe that the common assessments are better 
aligned with the curriculum and the pacing guides than the interim assessments. This contextual factor may present some 
challenges to the district's efforts to achieve a consistent measure of students' strengths and weaknesses across the district. 
In 2009-10, D istrict 2 administered interim assessments three times in most grades. 

Districts As of 2010, District 3 served close to 98,000 students in 155 schools and employed more than 6,000 teachers. 
The district's vision for data use is driven by nine broad organizational standards focused on collaborative and data-driven 
decision making. This district began implementing interim assessments in 2005 to gauge students' academic progress before 
the end-of-year state exam. The i nteri m assessment program in this district is different from those of other districts in several 
ways. First, the assessments include open-ended questions. As a result, significant effort is put into training staff on how 
to score these questions to ensure that the results are consistent from school to school. Second, the district created the data 
management system in-house, so it has more flexibility to modify the system to meet staff needs. 

This district currently operates under a school-based management model; therefore, individual schools have some degree 
of autonomy in how much they use data as a driving force for instructional changes and whether interim assessments are 
administered in their schools. Consequently, the understanding of interim assessments and how they can be used varies 
from school to school. The district has built the capacity to systematically monitor school-based activities through several 
structures, including the data- management system, district staff collaborations, subject- and grade-based planning, profes- 
sional development, and leadership meetings. In 2009-10, District 3 administered interim assessments up to seven times in 
some grades. 

District4 In 2010, District 4 served close to 24,000 students in 53 schools and had a staff of approximately 1,900 teachers. 
Interim assessments had been in use in the district for more than 10 years. The decision to adopt district interim assessments 
stemmed from a history of low student performance that had placed the district among the states' lowest performing. Initial- 
ly, teachers did not see the purpose or potential benefits of interim assessments; however, the district worked to encourage 
buy-in from the schools and, according to district representatives, has improved engagement with the interim assessment 
process over the past seven years. 
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The district’s management model includes significant oversight of the interim assessment process. The district described a 
culture of data use in which teachers understand the expectations for using data and are engaged in the interim assessment 
process. The district also emphasized collaborative relationships that allow teachers and principals to be part of the decision- 
making process about the interim assessments. Hiring practices at the administrative level are additional evidence of how 
District 4 has taken steps to create a more sustainable data-driven culture. Several district- based staff reported that experi- 
ence in using data was a requirement for their positions, suggesting a commitment to data use in the district. In 2009-10, 
District 4 administered interim assessments three times in most grades. 

School Sample 

A total of 193 elementary and middle schools across the four districts were included in the study. To gather a sample of 
schools that was representative of all elementary and middle schools in the participating districts, we conducted a stratified 
random sampling procedure within districts. 

B ased on i nitial power calculations, we sought a sample of 100 elementary schools and 80 middle schools per district (yield- 
ing approximately 800 elementary school teachers and 960 middle school teachers). We also wanted to over-sample by 5 
percent to account for attrition/nonresponse, resulting in a total of 107 elementary schools and 85 middle schools that we 
sought to invite to participate. For the elementary school sample, we simply divided the required number of schools based 
on the power analyses (107) by the number of districts (4), therefore seeking between 26 and 28 schools per district. For 
the middle school sample, we sought an uneven number of schools per district because one district had only nine middle 
schools. To accommodate this, we sampled a larger number of schools from another district. 

Stratified random sampling was conducted in each district to select the desired number of elementary and middle schools. 
Schools within district were stratified by (1) school type (elementary versus middle school), (2) percent economical ly disad- 
vantaged (i.e., percent eligible for the National School Lunch Program), and (3) percent ethnic/racial minority enrollment. 
The percentage of students served per school who were eligible for the school lunch program and were racial/ethnic minori- 
ties was gathered from the U .S. Department of Education's Common Core of Data. We used the median value on percent 
economically disadvantaged and percent minority for elementary and middle schools, within district, to create groups of 
schools that were either low or high on economic disadvantage or minority status. 3 Based on these groupings, schools were 
then assigned to one of six strata (e.g., Stratum 1 included elementary schools with high percent economic disadvantage and 
high percent minority enrollment). 

We then drew a simple random sample from each stratum where the number of schools chosen from each stratum was pro- 
portionate to the number of schools in that stratum. For example, if 60 percent of schools were in stratum 1, then 60 percent 
of the sampled schools were drawn from stratum 1. 


3 M edian values were calculated separately for elementary and middle schools; therefore, there were different cutoff scores for high versus low economic disadvantage 
and minority enrollment for elementary and middle schools within and between districts. 


Council of the Great City Schools • American Institutes for Research • Summer 2012 


RESEARCH DESIGN CONT’D 


Table 2.2. Number of Schools Sampled per District 


District 

Number of 
E lementary Schools 

Number of 
M iddle Schools 

District 1 

26 

21 

District 2 

27 

34 

District 3 

26 

21 

District 4 

28 

9 

Total 

107 

85 


Note. In District 4, all schools were invited to participate rather than a random stratified sample of schools. 


Teacher Sample 

Teachers in the sampled schools were invited to participate in the surveys if they taught reading or mathematics in grades 
4, 5, 7, or 8. 4 We selected these grade levels because the district survey indicated that most urban districts are especially 
focused on administering interim assessments in grades 3-8. The entire sample of teachers includes 1,581 elementary and 
middle grades teachers of reading or mathematics who were linked with students via classroom rosters. For more detailed 
information about the sample, please see Appendix C. 

E lementary School Teachers. Of the 1,581 teachers, 623 taught reading and/or mathematics to students in grades 4 and 5. 
Of the 623 elementary grade teachers, most (92 percent) taught both reading and mathematics; 2 percent taught mathematics 
only, and 6 percent taught reading only. Table 2.3 shows the demographic characteristics of the elementary school teachers 
included in the analytic sample, for whom demographic data were available. A majority of the elementary grade teachers 
who responded to the surveys were white (69 percent), and most were women (83 percent). Seventy-two percent of the 
sample had a master's degree or higher. The respondents had, on average, more than 11 years of experience teaching and 
6 years in their current schools, although there was a wide range in years of experience ranging from less than 1 year to 44 
years (SD = 8.67). 


4 In District 3, we sampled teachers within middle schools to attain the target sample size. In this large district, there were many more middle school teachers nested 
within the sample schools than were necessary based on our power calculations (633 versus 431). Therefore, a random sample of teachers within sampled schools was 
conducted whereby 68 percent of teachers within each school were randomly selected. Therefore, rather than including all seventh- and eighth-grade teachers within 
each sample middle school, a range from 14 to 29 teachers per school were invited to participate, based on the size of the school. The total number of teachers invited to 
participate in District 3 was 431. 
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Table 2.3. Demographic C haracteristics of Fourth- and Fifth-G rade Teacher Sample 



M athematics 


Reading 


Total 


Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Degrees obtained 







Below bachelor’s degree 

0 

0% 

0 

0% 

0 

0% 

Bachelor’s degree 

514 

100% 

538 

100% 

547 

100% 

Master’s degree 

314 

72% 

326 

72% 

332 

72% 

Educational specialist or 
professional diploma 

80 

25% 

83 

25% 

83 

25% 

C erti fi cate of A dvanced 
Graduate Studies 

63 

21% 

67 

22% 

68 

22% 

Doctorate or professional 
degree (Ph.D., Ed.D.) 

9 

3% 

9 

3% 

9 

3% 

Gender (Female) 
Race 

433 

84% 

452 

84% 

459 

83% 

White 

354 

69% 

371 

70% 

376 

69% 

Black 

75 

15% 

79 

15% 

82 

15% 

Asian 

6 

1% 

6 

1% 

6 

1% 

American Indian 

4 

1% 

4 

1% 

4 

1% 

M ulti racial 
Ethnicity 

12 

2% 

13 

2% 

13 

2% 

Latino/Latina 

59 

12% 

60 

11% 

61 

11% 

M issing race/ethnicity 
information 

77 

13% 

79 

13% 

81 

13% 


Note. Teachers who taught both subjects are included in both the Mathematics and Reading columns. The numbers 
in these columns do not add up to the numbers in theTotal column because of the overlap. Percentages reported in 
table 2.3 are based on teachers for whom we had complete demographic data. 


M iddle School Teachers. The middle school sample included 958 teachers; 473 (49 percent) of these respondents taught 
middle school mathematics, and 552 (58 percent) taught middle school reading. Only 67 (7 percent) of the 958 middle 
school teachers taught both mathematics and reading. Table 2.4 presents the demographic characteristics of the middle 
school teachers, for whom demographic data were available. Similar to the elementary school respondents, a majority of 
the middle school teachers who responded were white (71 percent), and most (78 percent) were women. On average, the 
middle school respondents had an average of more than 11 years of teaching experience, ranging from less than 1 year to 
44 years (SD = 8.91). With respect to degrees earned, all but one teacher in the sample had at least a bachelor's degree, and 
72 percent had a master's degree. 
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Table 2.4. Demographic Characteristics of Seventh- and Eighth-Grade Teacher Sample 



M athematics 


Reading 


Total 


Number 

Percentage 

N u m ber Percentage 

Number 

Percentage 

Degrees obtained 







Below bachelor’s degree 

1 

<1% 

0 

0% 

1 

<1% 

Bachelor’s degree 

425 

> 99% 

482 

> 99% 

849 

> 99% 

Master’s degree 

Educational specialist or 
professional diploma 

274 

79 

72% 

30% 

321 

83 

73% 

28% 

551 

150 

72% 

29% 

Certificate of Advanced 
Graduate Studies 

51 

20% 

60 

21% 

104 

21% 

Doctorate or professional 
degree (Ph.D., Ed.D.) 

6 

3% 

13 

5% 

19 

4% 

Gender (Female) 
Race 

306 

71% 

412 

85% 

671 

78% 

White 

299 

71% 

338 

72% 

597 

71% 

Black 

43 

10% 

61 

13% 

97 

12% 

Asian 

9 

2% 

3 

1% 

12 

1% 

American Indian 

1 

0% 

2 

0% 

3 

0% 

M ulti racial 
Ethnicity 

13 

3% 

12 

3% 

23 

3% 

Latino/Latina 

57 

14% 

55 

12% 

103 

12% 

Missing race/ethnicity 
information 

51 

11% 

81 

15% 

123 

13% 


Note. T eachers who taught both subjects are included in both the M athematics and Reading columns. Thenumbers 
in these columns do not add up to the numbers in the T otal column because of the overlap. Percentages reported in 
table 2.4 are based on teachers for whom we had complete demographic data. 


Principal Sample 

All principals (and assistant principals, where appropriate) of the participating schools were asked to complete the surveys. 
The sample included a total of 212 principals, with 124 principals from elementary schools, 86 principals from middle 
schools, and 2 principals from K-8 schools (who were considered principals of middle grades for this study). Table 2.5 
presents the demographic characteristics of principals for whom demographic data were available. Overall, 76 percent of 
the principals and assistant principals in the survey sample were women— from elementary schools, the proportion of fe- 
male principals was 84 percent, and from middle schools, 65 percent. Forty-eight percent of elementary principals and 64 
percent of middle school principals were white, and 40 percent of elementary and 23 percent of middle school principals 
wereAfrican American. 

All of the elementary and middle school principals and assistant principals had at least a master's degree, and 23 percent of 
the elementary school principals and 17 percent of the middle school principals had a Ph.D. or Ed.D. In terms of experience, 
the overall average years of experience for principals were 14 years as teachers and 11 years as principals. Table 2.5 presents 
the demographic characteristics of the principals included in the analytic sample. 
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Table 2.5. Demographic C haracteristics of Principal Sample 



E lementary G rades 

Middle Grades 


Total 


Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Degrees obtained 







Below bachelor’s degree 

0 

0% 

0 

0% 

0 

0% 

Bachelor’s degree 

103 

100% 

75 

100% 

178 

100% 

Master’s degree 

105 

100% 

75 

100% 

O 

CO 

I — 1 

100% 

Educational specialist or 
professional diploma 

37 

51% 

30 

58% 

67 

54% 

C erti fi cate of A dvanced 
Graduate Studies 

24 

39% 

14 

31% 

38 

36% 

Doctorate or professional 
degree (Ph.D., Ed.D.) 

13 

23% 

7 

17% 

20 

21% 

Gender (Female) 
Race 

87 

84% 

49 

65% 

136 

76% 

White 

48 

48% 

48 

64% 

96 

55% 

Black 

40 

40% 

17 

23% 

57 

32% 

Asian 

2 

2% 

1 

1% 

3 

2% 

American Indian 

1 

1% 

1 

1% 

2 

1% 

M ulti racial 

4 

4% 

2 

3% 

6 

3% 

Ethnicity 







Latino/Latina 

6 

6% 

6 

8% 

12 

7% 

Missing race/ethnicity 
information 

23 

19% 

13 

15% 

36 

17% 


Note. Percentages reported in table 2.5 are based on principals for whom we had complete demographic data. 


Student Sample 

The student sample included 61,798 students across the four districts, of which 14,974 were in grades 4 or 5 and 46,824 
were in grades 7 or 8. Their demographics are shown in Table 2.6. The student sample was 34 percent white, 22 percentAf- 
rican American, and 36 percent Hispanic. Sixty-five percent of the elementary grade students and 59 percent of the middle 
school students were eligible for the National School Lunch Program. Fifteen percent of students in the elementary grades 
sample and 12 percent in the middle grades sample received special education services. Thirteen percent of the elementary 
grades sample and 11 percent of the middle grades sample were E nglish language learners. 
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Table 2.6. Demographic C haracteristics of Student Sample 



E lementary G rades 

M iddle G rades 


Total 


Number 

Percentage 

Number 

Percentage 

Number 

Percentage 

Gender (Female) 
Race 

6,901 

49% 

22,420 

49% 

29,321 

49% 

White 

9,680 

31% 

29,634 

35% 

39,314 

34% 

Black 

3,951 

28% 

9,001 

20% 

12,952 

22% 

Asian, Native Hawaiian, or 
Pacific Islander 

709 

5% 

2,966 

6% 

3,675 

6% 

American Indian 
Ethnicity 

229 

2% 

557 

1% 

786 

1% 

Latino/Latina 

4,724 

33% 

17,027 

37% 

21,751 

36% 

Missing race/ethnicity 
information 

857 

6% 

1,115 

2% 

1,972 

3% 

Received free or reduced-price 
lunch 

9,603 

65% 

26,459 

59% 

36,062 

61% 

Received special education 

2,244 

15% 

5,334 

12% 

7,578 

13% 

services 




ELL status 

1,935 

13% 

4,908 

11% 

6,843 

12% 


Note. Percentages reported in table 2.6 are based on students for whom we had complete demographic data. 


Measures 

To examine the relationships among the key dimensions of data use to test the hypothesized links in the theory of action, 
we measured teachers' and principals' data-use practices and perceptions using surveys administered three times over the 
course of the 2009-10 school year. Our analysis focused on the relationships between the four key dimensions of data use 
(Context Supports for Data Use Working with Data, and Instructional Responses) and student achievement on the state 
assessments in reading and mathematics. We also examined the links between perceived Barriers to Data Useand student 
achievement in both subjects. 

Surveys of Teachers' and School Principals' Data-Use Practices and Perceptions 

To gather data on data use, we surveyed teachers and school principals about their data-use practices at three points during 
the 2009-10 school year. The surveys were administered online and measured the key dimensions of data use. 

The surveys were designed to measure each of the key dimensions included in the theory of action. To develop the surveys 
of teacher and principal data-use practices and perceptions, researchers conducted an initial scan of more than 40 previously 
used surveys that measured some aspect of data use (general or specific). Researchers then narrowed the item pool to the 
most relevant, reliable, and valid measures of data use— 229 items from 22 source surveys. Items from the source surveys 
were examined for content and psychometric properties. The source surveys included instruments specifically developed 
to measure data use, such as the online Ohio Department of Education (ODE) School Administrator Survey, 5 as well as 
surveys that had a broader purpose, such as those used for the National Longitudinal Study of No Child Left Behind (U .S. 
Department of Education, 2010b). 


5 The ODE survey is available online at, (www.ode.state.oh.us/GD/Templates/Pages/ODE/ODEDetail.aspx?page=3&TopicRelationID=1222&ContentID=57327&Cont 
ent=94770). 
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After a pool of possible items was selected, the items were then mapped back onto the theory of action to ensure adequate 
coverage of each key dimension of data use. There were many available items addressing many of the key dimensions of 
data use (e.g., issues of staffing/human resources related to data use). However, there was a dearth of items addressing other 
key dimensions (e.g., instructional responses). To further develop the surveys for the purpose of the study, existing item 
sets were modified and developed to sufficiently measure all the key dimensions, including I nstructional Responses Items 
included in the survey were mostly five-point Likert-type items but also included frequency count questions to measure 
teachers' and principals' data-use practices. 

In summer 2009, an in-depth cognitive laboratory process to pilot the items with 17 teachers and principals in the Washing- 
ton, D.C., area was conducted. Fourteen of the cognitive laboratory participants were teachers— four in elementary schools 
and ten in middle schools; two were principals, and one was a vice principal. Using ThinkAloud techniques, participants 
answered the questions on the survey while saying everything they were thinking. This process allowed for an investigation 
into the cognitive processes that teachers and principals employ when thinking about the survey items. In addition, cognitive 
laboratories provide information about factors other than data-use practices that may have an effect on teachers' responses 
to the items. This technique allows participants to talk through any possible issues with survey items, including confusing 
wording, content issues, vocabulary that may not be understood, or confusing response structures. We then refined the sur- 
vey items on the basis of the feedback provided by the teachers and principals during this process. 

Survey items from the data-use surveys were tailored to measure the four key dimensions of data use— including Context 
Supports for Data Use Working with Data, and Instructional Responses- as well as the key Barriers to Data Use previ- 
ously noted in the literature on data-driven decision making. The following sections provide descriptive information about 
the survey items used to measure each construct. The teacher and principal surveys included many common items, with 
some unique items on each. Also, some survey items for both principals and teachers were subject specific for reading or 
mathematics instruction. M ost items were administered in all three waves; however, some item sets were included in only 
one wave. For example, questions about data coaches were included only in the second wave. Example survey items used to 
measure the dimensions of data use among teachers and principals are provided in A ppendix B (Table B .3). 

Contact The surveys included items that measured assessment/instructional context and state, district, and 
school data culture. Survey items measuring assessment/instructional context asked teachers and principals about 
different types of assessments and the alignment of interim assessments with other assessments. State, district, 
school data culture were measured with items gauging state, district, and school leader supportiveness of and 
emphasis on data-use practices. There were 36 items on the teacher survey and 32 items on the principal survey 
that measured constructs related to Context 

Sup p orts for Data Use This key dimension was measured with items representing three constructs: data 
infrastructure, organizational supports, and staffing and human resources. Survey items asked teachers and 
principals about the data system, the support they receive from the district and school around data use, and the 
professional development training they receive to use interim assessment data. Teachers' perceptions of Supports 
for Data Usewere measured with 48 survey items; the principal survey had 51 items. 

Wbridngwith Data The key dimension of W)rking with Data was measured with 42 items that measured aspects 
of teachers' and principals' individual attention to data and collaboration around data. Both teachers and principals 
were asked to report the amount of time they spent individually examining student data. To measure collaboration 
in the review of data among key actors— including teachers, principals, data coaches, parents, and students— the 
survey assessed the frequency of collaborative interactions about data, including, specifically, the number and 
types of interactions between teachers and principals; teachers and instructional coaches; and teachers, principals, 
and parents. 
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Instructional Responses Instructional Responses were measured with survey items that asked teachers and 
principals about changes made in the classroom or school in response to their review and interpretation of 
interim assessment results. On the teacher survey, 26 items represented four main constructs: adjust lesson plans, 
establish/adjust groupings, provide supplemental resources to targeted students, and change scope and sequence. 
On the principal survey, 24 items related to Instructional Responses that represented four constructs: adjust 
lesson plans, establish/adjust groupings, provide supplemental resources to targeted students, and school-level 
instructional responses. Survey items measuring school-level Instructional Responseswere asked of principals 
only, in reference to their perceptions of the influence of interim assessment results on schoolwide issues 
such as evaluating school initiatives or programs and identifying professional development needs. 

Barriersto Using Dala Perceptions of Barriers to Data Use were measured with survey items that asked teachers 
and principals about situational factors that may prevent them from using interim assessment data. For example, 
survey items asked principals and teachers if inadequate professional development or curriculum pacing pressures 
interfered with their ability to use interim assessment results. For teachers, Barriersto Data Use were measured 
with 15 survey items; for principals, there were 13 survey items. All survey items measuring Barriersto Data 
Use are Likert-type items. 

Data-UseScaleConstruction 

Survey items measuring the constructs were combined into scale scores to provide a single measure for each of the four 
key dimensions and barriers. The survey items within each scale varied in response formats, including Likert items, binary 
response items, and frequency counts (e.g., number of hours spent reviewing data). Therefore, we standardized the item 
responses to create scale scores. First, each item response was standardized within wave using z-score standardization with 
a mean of zero and a standard deviation of one. Items were standardized separately for the teacher and principal samples. 
Next, each standardized item was averaged across the three administration waves to incorporate responses from all available 
waves of data collection. 

Finally, the standardized item responses averaged across waves were used to create the five scales — one for each of the four 
key dimensions plus one representing barriers. Scale averages were calculated for each survey respondent who responded to 
at least 50 percent of the survey items within the scale. For example, the Context scale score for teachers contains 36 survey 
items. Eighteen completed survey items were required to compute a scale score for Contextfor any participating teacher. 

The internal consistency reliability was moderate to high for all scales on both the teacher and principal surveys. Cronbach's 
alpha reliability coefficients ranged from 0.74 to 0.97 for the teacher scales, and from 0.76 to 0.97 for the principal scales, 
indicating adequate internal reliability. For a detailed description of the number of items per scale and individual reliability 
coefficients, see Tables B.l and B.2 in Appendix B. 

Measures of Teacher and Principal Characteristics 

The teacher and principal surveys contained items on respondents' background and demographic characteristics, including 
education level, race/ethnicity, and gender. The surveys also collected information about teaching experience, including the 
total number of years teaching and number of years teaching at the current school, for both teachers and principals. Princi- 
pals were additionally asked to report the total number of years of administrator experience and the number of years they 
had served as a principal at their current schools. 
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Student I nformation 

District-provided student data were shared through secure file transfer protocol sites established between AIR and each par- 
ticipating district. Data were collected for a total of 86,837 students across the four districts. These data included individual 
student data, such as achievement and demographics, as well as student-teacher rosters that connected students with their 
teachers and classrooms. 

Student C haracteristics and Demographics. Each district provided demographic data for students in grades 4, 5, 

7, and 8 during the 2009-10 school year. Demographic information available in the district administrative records 
included gender, race/ethnicity, free or reduced-price lunch eligibility, special education services eligibility, 
and English language learner (ELL) status. 

Student Achievement in Mathematics and Reading. The district-provided student records data included state as- 
sessment data in mathematics and reading from the two years prior to the study (spring 2008 and 2009) and for the 
year of the study (spring 2010) for all students in grades 4, 5, 7, and 8 enrolled in the participating schools in the 
2009-10 school year. The assessments and the possible range of scores were different in each district, as they are 
each in different states. For example, District 1 scores ranged from 0 to 999; District 2 scores ranged from 100 to 
500; and D i strict 4 scores ranged from 0 to 600. 1 n these three districts, the scale scores were the same across grades; 
however, District 3 has grade-specific scale scores based on an 80-point scale system (i.e., grade 4 scores ranged 
from 400 to 480; grade 5 scores ranged from 500 to 580; grade 7 scores ranged from 700 to 780; and grade 8 scores 
ranged from 800 to 880). Because each state assessment measured student achievement differently, we standardized 
the student achievement data within state and grade level. 

Classroom Rosters (Student-Teacher Assignments). All districts also provided classroom roster data for math- 
ematics and reading classes in grades 4, 5, 7, and 8 that listed the students assigned to each teacher in each class. 
Specifically, the rosters included course name, teacher name, teacher ID, and student IDs for all students in the class. 

Data Collection Procedures 

Teacher and Principal Survey Administration 

Survey data collection occurred three times over the course of the 2009-10 school year. The surveys were administered 
online using SurveyM onkey® in fall 2009, winter 2010, and spring 2010. Each survey administration was timed to begin 
between 7 and 14 days after each district's interim assessments were administered in an effort to obtain accurate measures 
of teachers' and school principals' use of student data. In the surveys, respondents were asked to respond to items with 
respect to the latest round of interim assessments. Table 2.7 displays the specific start and end dates in each district. The 
surveys remained open for approximately 3 weeks. 


Council of the Great City Schools • American Institutes for Research • Summer 2012 


RESEARCH DESIGN CONT’D 


Table 2.7. Survey Administration Dates by District 



District 1 

District 2 

District 3 

District 4 

Wavel 

Date survey opened 

11/16/2009 

11/30/2009 

11/3/2009 

11/18/2009 

Date survey closed 

12/8/2009 

12/18/2009 

11/30/2009 

12/9/2009 

Wave 2 

Date survey opened 

2/9/2010 

3/1/2010 

3/1/2010 

3/17/2010 

Date survey closed 

3/2/2010 

3/29/2010 

3/29/2010 

4/14/2010 

Wave 3 

Date survey opened 

5/3/2010 

5/12/2010 

5/10/2010 

5/18/2010 

Date survey closed 

5/25/2010 

6/4/2010 

6/4/2010 

6/14/2010 


An initial e-mail was sent to the district e-mail address of every teacher in the sample, inviting them to participate in the 
survey. 6 E-mail reminders were sent weekly thereafter to nonrespondents. Other follow-up strategies included postcard 
reminders sent approximately one week after the survey opened and follow-up calls made one week before the end of the 
survey. U pon completion of the online survey, teachers received a $25 gift card. To further encourage respondents to par- 
ticipate in the second and third waves of administration, those who completed the surveys also were entered into a raffle for 
an additional $100 gift card. 7 

Survey Response Rates 

Detailed information about both the teacher and principal response rates by wave and by district are presented below. In 
general, the percent of respondents invited to participate that responded to at least one of the three waves of online surveys 
was 83 percent for teachers and 87 percent for principals. 

Teachers. Across the four participating districts, 2,248 teachers were invited to participate in the online survey. Response 
rates per survey wave ranged from 64 percent to 69 percent (seeTable 2.8). 


Table 2.8. Teacher Survey Response Rates, by Wave and District 




Wavel 



Wave 2 



Wave 3 


District 

1 nvited 

Responded 

Response 

Rate 

1 nvited 

Responded 

Response 

Rate 

1 nvited 

Responded 

Response 

Rate 

District 1 

643 

470 

73% 

643 

472 

73% 

643 

446 

69% 

District 2 

689 

496 

72% 

704 

454 

64% 

706 

444 

63% 

District 3 

624 

398 

64% 

620 

407 

66% 

619 

389 

63% 

District 4 

273 

169 

62% 

273 

176 

64% 

271 

162 

60% 

Total 

2,229 

1,533 

69 % 

2,240 

1,509 

67 % 

2,239 

1,441 

64 % 


To include as many teachers as possible in the final analyses, data were retained if a teacher responded to at least one wave 
of the survey. A total of 1,855 teachers completed at least one wave of the data-use survey during the 2009-10 school year, 
for an overall response rate of 83 percent. Table 2.9 displays the response rates for the sample of teachers who responded to 
at least one of the three waves of survey administration, by district. 


6 One district did not share teacher e-mail addresses; therefore, the online survey invitation and reminders were sent by the school district rather than the study team. 

7 A total of 4,483 $25 gift cards were distributed to survey respondents, and a total of eight $100 gift cards were distributed to lottery winners, accounting for $112,875 
spent on incentives designed to boost teacher and principal response rates. 
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Table 29. Pmxntagecf Teachers Responding to 
at Least One Wave of the Survey, by District 


District 

Invited 

Participated in 
at least one 

% Respond ng to at least 
one wave 



wave 

District 1 

643 

561 

87% 

District 2 

707 

587 

83% 

District 3 

624 

488 

78% 

District 4 

274 

219 

80% 

Total 

2,248 

1,855 

83 % 


Survey responses from teachers who could not be linked to individual students via classroom rosters (e.g., if they taught 
electives or supplemental courses that did not have classes with English language arts or mathematics titles) were excluded. 
Following these exclusions, we had an analytic sample of 1,581 teachers across the four districts. Proportionately, 30 per- 
cent of the analytic teacher sample was from District 1, 32 percent was from District 2, 26 percent was from District 3, and 
12 percent was from District 4. 

Principals Across the four districts, 244 principals or assistant principals were invited to participate. Response rates by 
wave ranged from 64 percent to 85 percent (seeTable 2.10). 

Table 2 IQ. Principal Sirvey Response Rates* by Wave and District 




Wavel 



Wave 2 



Wave 3 


District 

Invited 



Invited 


Response 

Rate 

Invited 



Response 

Rate 

Response 

Rate 

Responded 

Responded 

RespaTsec! 

District 1 

47 

36 

77% 

47 

34 

72% 

47 

35 

74% 

District 2 

63 

48 

76% 

65 

34 

52% 

66 

44 

67% 

District 3 a 

47 

40 

85% 

49 

34 

69% 

48 

37 

77% 

D i stri ct 4 b 

39 

29 

74% 

i — 1 

OO 

54 

67% 

i — 1 

OO 

56 

69% 

Total 

196 

153 

78% 

242 

156 

64% 

242 

172 

71% 


Afore. The increase in the number of invited principals in District 4 between Wave 1 and Waves 2 and 3 represents the 
inclusion of assistant principals in Waves 2 and 3. This change was made because assistant principals were reported to 
be more involved than principals in the interim assessment data process in this district. 


a A cross the three waves of data collection, 50 principals from District 3 were invited to take the survey. Between the 
first wave and the third wave, two principals retired, and three more principals were invited to take the survey. 
b One principal in District 4 was the principal of two schools. 


As with the teachers, principals were retained in the survey sample if they completed a survey for at least one wave of data 
collection. Table 2.11 displays the principal response rates across any of the three waves of surveys. Of the 244 principals 
or assistant principals invited to participate, 212 (87 percent) completed one or more of the three online surveys. Of the total 
respondents, 124 were elementary school principals or assistant principals, 86 were middle school principals or assistant 
principals, and 2 were principals at K-8 schools and included in the middle school sample. 
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Table 2.11. Percentage of Principals Responding to at Least One Wave of the Survey, 

by District 


District 

1 nvited 

Responded 

Response Rate 

District 1 

47 

42 

89% 

District 2 

66 

56 

85% 

District 3 

50 

45 

90% 

District 4 

81 

69 

85% 

Total 

244 

212 

87% 


Linking Teacher Survey Data to Student Achievement 

To implement our analysis strategy, it was necessary to link teachers' survey data to their own students' achievement data. 
This linking process occurred in two stages: We linked teachers’ survey responses with their district-provided identifiers; 
then, we linked teachers with their students using the classroom rosters provided by the district. 

One complication with linking the students to their teachers was that some students were enrolled in multiple reading or 
mathematics courses. For example, a student may be enrolled in both a reading class and a language arts class taught by 
different teachers; therefore, that student was associated with multiple teachers in the "reading teacher” survey sample. 
Five percent of the students i n the sample were enrolled in multiple reading classes, and 5 percent were enrolled in multiple 
mathematics classes. Enrollment in multiple mathematics or multiple reading classes was more prevalent in middle grades 
in Districts 3 and 4 where the percentage of students enrolled in two or more reading classes was as high as 13 percent 
and 12 percent, respectively. To address this problem, we randomly selected unique student- teacher links to represent the 
teacher-student mathematics and reading relationships. To do this, we first identified each unique student-teacher link in the 
four district data files. After the files were cleaned of any duplicate links, each student-teacher link became a row in the data 
file. Next, a random number generator was used to create lists of numbers associated with each student-teacher link. The file 
was then sorted by a student identifier and the random number. To randomly select one student-teacher link per student per 
subject, only the first student-teacher link was retained in the data file, thus randomly selecting one teacher to represent each 
student's mathematics or reading educator. 

When the student-teacher links were established, we created one large analytic file that included teachers’ and principals’ 
survey responses reflecting their data-use practices and perceptions and all relevant data for their own students including 
background characteristics and multiple years of achievement data. 

Analytic Strategy 

To reiterate the focus of the study, we posed two broad research questions regarding data-driven instruction: 

1. What are the relationships between teachers' data-use practices and perceptions and their 
students' achievement? 

2. What are the relationships between schools' policies, practices, and resources for data-driven 
instruction and student achievement? 


46 CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


To address these research questions, we began with a set of descriptive analyses examining levels of data use in elementary 
and middle schools and for teaching of reading and mathematics. We also examined the simple correlations among the four 
dimensions of data use, barriers, and student achievement. 

Based on the exploratory descriptive analyses, we then conducted two sets of analyses to examine how teacher- and school- 
level data-use practices and perceptions were related to student achievement. Because students were nested within teachers, 
which were in turn nested within schools, a multilevel framework was used for both sets of analyses. 

The first set of analyses used structural equation modeling (SEM) to examine the relationship between a global sense of data 
use and student achievement. The second set used hierarchical linear modeling (HLM ) to examine the unique I inks between 
each dimension of data use and student achievement, independent of the other dimensions. 
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RESULTS 


RESULTS 


This chapter presents the results of analyses that tested whether and how data-use practices and perceptions are linked to 
student achievement. First, we present descriptive statistics and bivariate correlations among the key variables. Then, we 
provide the results of the statistical analysis organized by the research questions. 

Descriptive Statistics 


Descriptive statistics for teacher and principal data-use variables are presented in Table 3. 1-3. 4. Because data-use scores 
were standardized, all data-use scale scores have a mean of zero and a standard deviation of one, where higher scores indi- 
cate higher levels of data use. 

M ean C omparisons of Data U se 

In general, scores on the data-use scales were higher in elementary grades than in middle grades. In both reading and mathe- 
matics, elementary grade teachers reported higher levels of data use than did middle school teachers. To compare elementary 
and middle grades teachers' reported data-use practices and perceptions, we conducted independent-samples t-tests with 
each of the four key dimensions of data use and barriers as the dependent variables. These tests of difference were conducted 
separately for mathematics and reading teachers. As shown in Table 3.1, elementary grade reading teachers reported higher 
levels of data use than did middle school reading teachers in terms of Context, Supports for Data Use, and Working with 
Data. No significant differences on Instructional Responses or perceived Barriers were detected. 


Table 3.1. E lementary and Middle Grades Teachers' 
Reported Key Dimensions of Data Use in Reading 



E lementary G rades 
Reading 
Mean (SD) 

M iddle G rades 
Reading 
Mean (SD) 

Mean Difference 

(Elementary - Middle) 

Context 

0.02 (0.38) 

-0.04 (0.44) 

0.06** 

Supports for data use 

0.02 (0.50) 

-0.07 (0.47) 

0.09** 

Working with data 

0.07 (0.46) 

-0.03 (0.47) 

0.10*** 

Instructional responses 

0.06 (0.60) 

0.02 (0.66) 

0.04 

Barriers 

0.02 (0.44) 

-0.03 (0.50) 

0.05 


Note. **p< .01, ***p < .001, two-tailed. M eans were calculated across the entire sample of schools using 
standardized scores with a mean of 0 and a standard deviation of 1. 


There also were significant differences between elementary and middle grade mathematics teachers on two of the five data- 
use variables. Elementary mathematics teachers reported higher levels of Wirking with Data and Instructional Responses 
than did middle school mathematics teachers. There was no significant difference between elementary and middle school 
mathematics teachers on Context Supports for Data, or Barriers(seeTable 3.2). 


Table 3.2. E lementary and M iddle Grades Teachers' 
Reported Key Dimensions of Data Use in Mathematics 



E lementary G rades 
Reading 
Mean (SD) 

M iddle G rades 
Reading 
Mean (SD) 

Mean Difference 

(Elementary — Middle) 

C ontext 

0.03 (0.37) 

0.00 (0.40) 

0.03 

Supports for data use 

0.01 (0.49) 

-0.03 (0.48) 

0.04 

Working with data 

0.08(0.46) 

-0.05 (0.47) 

013*** 

Instructional responses 

0.06(0.59) 

-0.06 (0.64) 

0.12** 

Barriers 

0.03 (0.44) 

0.02 (0.46) 

0.01 


Note. **p < .01, ***/? < .001, two-tailed. M eans were calculated across the entire sample of schools using 
standardized scores with a mean of 0 and a standard deviation of 1. 
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We saw similar significant differences among principals, where principals in elementary grades had higher scores on Con- 
text and Supports for Data Use than did those in middle grades. Also, principals in elementary grades reported lower levels 
of perceived Barriers than those in middle grades. Scores on Working with Data and Instructional Responses were not sig- 
nificantly different among elementary and middle school principals (see Tables 3.3 and 3.4). 


Table 3.3. E lementary and M iddle G rades Principals' 
Reported Key Dimensions of Data Use in Reading 



E lementary 
Reading 
Mean (SD) 

Middle Reading 
Mean (SD) 

Mean Difference 

(Elementary - Middle) 

Context 

0.02 (0.38) 

-0.07 (0.36) 

0.09** 

Supports for data use 

0.04 (0.40) 

-0.12 (0.40) 

0.16* 

Working with data 

-0.01 (0.56) 

-0.12 (0.45) 

0.11 

Instructional responses 

-0.05 (0.64) 

-0.09 (0.57) 

0.04 

Barriers 

-0.06 (0.38) 

0.22 (0.40) 

-0.28** 


Note. *p < .05, **p< .01, two-tailed. M eans were calculated across the entire sample of schools using standardized 
scores with a mean of 0 and a standard deviation of 1. 


Table 3.4. E lementary and M iddle G rades Principals' 
Reported Key Dimensions of Data Use in Mathematics 



E lementary 
Reading 
Mean (SD) 

Middle Reading 
Mean (SD) 

Mean Difference 

(Elementary - Middle) 

Context 

0.03(0.37) 

-0.07 (0.36) 

0.10* 

Supports for data use 

0.05 (0.41) 

-0.11 (0.40) 

0.16* 

Working with data 

0.00 (0.56) 

-0.12 (0.44) 

0.12 

Instructional responses 

-0.04(0.65) 

-0.08 (0.56) 

0.04 

Barriers 

-0.07 (0.40) 

0.23(0.39) 

-0.30*** 


Note. *p < .05, ***p < .001, two-tailed. Means were calculated across the entire sample of schools using 
standardized scores with a mean of 0 and a standard deviation of 1. 
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B ivariate C orrelations 

In addition to observing descriptive patterns between elementary and middle grades data-use practices and perceptions, we 
also examined correlations among the data-use scales as measured by our surveys. 

Bivariate Correlations Between Teacher Data-Use Variables 

As seen in Tables 3.5- 3.8, the four key dimensions— Context, Supports for Data Use Working with Data, and Instructional 
Responses- all were positively correlated in each analytic sample. These scales all were negatively correlated with teacher- 
reported Barriers, with correlations ranging from -0.09 to -0.39. All the positive correlations were statistically significant. 
Notably, the correlations between teacher- reported Working with Data and Instructional Responses were greater than 0.75 
in all four analytic samples. 

Table 3.5. Bivariate Correlations Among Data-Use Scales 
for E lementary G rades M athematics Teachers (N = 593) 


T eacher Data-Use Scales 

Context 

Supports for 
Data Use 

Working Instructional 
W ith D ata R esponses 

Barriers 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.49** 

0.39** 

0.41** 

-0.34** 

0.64** 

0.45** 

-0.34** 

0.77** — 

-0.17** -0.09 


Note. **p < .01, two-tailed. 





Table 3.6. Bivariate Correlations Among Data-Use Scales 
for E lementary School Reading Teachers (N = 614) 


T eacher Data-Use Scales 

Context 

Supports for 
Data Use 

Working Instructional 
W ith D ata R esponses 

Barriers 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.51** 

0.38** 

0.41** 

-0.33** 

0.65** 

0.47** 

-0.35** 

0.76** — 

-0.20** -0.13** 


Note. **p < .01, two-tailed. 





T able 3.7. B ivariate C orrelations of Data-U se Scales 
for M iddle School M athematics Teachers (N = 471) 


T eacher Data-Use Scales 

Context 

Supports for 
Data Use 

Working Instructional 
W ith D ata R esponses 

Barriers 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.63** 

0.54** 

0.48** 

-0.35** 

0.68** 

0.54** 

-0.35** 

0.80** — 
-0.14** -0.10* 



Note. *p < .05, **p < .01, two-tailed. 
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T able 3.8. B ivar iate C or relations of Data-U se Scales 
A mong M iddle School R eading T eachers (N = 532) 


T eacher Data-Use Scales 

Context 

Supports for 
Data Use 

Working 
With Data 

Instructional 

Responses 

Context 

— 




Supports for data use 

0.63** 

— 



Working with data 

0.58** 

0.68** 

— 


Instructional responses 

0.56** 

0.58** 

0.80** 

— 

Barriers 

-0.37** 

-0.39** 

-0.19** 

-0.13** — 


Note. **p < .01, two-tailed. 


Bivariate Correlations Between Principal Data-U se Variables 

Correlations among principal-reported data-use variables followed similar patterns, where Instructional Responses were 
highly and positively correlated with Vtorking with Data across the four analytic samples (with correlation coefficients close 
to or greater than 0.80). Context Supports for Data Use and Working with Data were moderately to highly correlated (from 
0.41 to 0.70). Barrierswere negatively correlated with all four variables in elementary schools (ranging from -0.28 to -0.50 
for both elementary and middle schools) but were not correlated with Supports for Data Usein middle schools (seeTables 
3.9-3.12). 


T able 3.9. B ivar iate C or relations of Data-U se Scales 
Among Principals in E lementary School M athematics Sample (N = 102) 


Principal Data-U se Scales 

Context 

Supports for 
Data Use 

Working 
With Data 

Instructional H 

Responses 

Context 

— 




Supports for data use 

0.59** 

— 



Working with data 

0.50** 

0.70** 

— 


Instructional responses 

0.53** 

0.61** 

0.85** 

— 

Barriers 

-0.40** 

-0.50** 

-0.38** 

-0.37** — 


Note. **p < .01, two-tailed. 


Table 3.10. Bivariate Correlations of Data-Use Scales 
Among Principals in E lementary School Reading Sample (N = 101) 


Principal Data-U se Scales 

Context 

Supports for 
Data Use 

Working 
With Data 

Instructional B . 
Responses Barners 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.56** 

0.49** 

0.50** 

-0.35** 

0.69** 

0.59** 

-0.47** 

0.84** 

-0.36** 

-0.36** — 


Note. **p < .01, two-tailed. 
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T able 3.11. B ivar iate C or relations of Data-U se Scales 
Among Principals in M iddle School M athematics Sample (N = 76) 


Principal Data-Use Scales 

Context 

Supports for 
Data Use 

Working 
With Data 

Instructional 

Responses 

Barriers 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.50** 

0.41** 

0.47** 

-0.37** 

0.65** 

0.55** 

-0.20 

0.80** 

-0.31* 

-0.34* 


Note. *p < .05, **p < .01, two-tailed. 





Table 3.12. Bivariate Correlations of Data-Use Scales 
Among Principals in M iddle School Reading Sample (N = 75) 


Principal Data-Use Scales 

Context 

Supports for 
Data Use 

Working 
With Data 

1 nstructional 
Responses 

Barriers 

Context 

Supports for data use 
Working with data 
Instructional responses 
Barriers 

0.51** 

0.39** 

0.47** 

-0.37** 

0.64** 

0.55** 

-0.20 

0.80** 

-0.28* 

-0.33* 



Note. *p < .05, **p < .01, two-tailed. 


For both teachers and principals, the scales measuring Working with Data and Instructional Responses were particularly 
strongly related, with correlation coefficients greater than r = 0.75 for the teacher samples and 0.80 for the principal samples. 
This observation led us to consider whether these aspects of data use — defined as distinct in the theory of action (Exhibit 
1.1)— are, in fact, too similar to separate, at least as measured by the survey instrument we used. As described in the next 
sections, we combined Vtorking with Data and Instructional Responses for some of the analyses. 

M ore information about the implications of the high correlations among the data-use variables (also known as multicol- 
I i nearity) is provided i n A ppendix D. 

A nalyses U sed to Test the R esearch Q uestions 


To test our research questions about the links between data-use practices and student achievement, we conducted two types 
of analyses. These are described in the next sections. 

1. We used structural equation modeling to test the relationship between student achievement and General 
Data Use- a combination of all the data-use variables. 

2. We used hierarchical linear modeling to test the relationship between student achievement and each data- 
use variable separately. 
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Structural E quation M odeling (SE M ) 

We used SE M to examine how one large construct of overall data use was related to student achievement at both the teacher 
and school levels. In this statistical approach, latent variables are created that combine information from multiple scales 
from the surveys. Although observed variables are directly measured (such as with a survey, observation, or interview), 
latent variables represent underlying constructs that are measured using multiple observed variables. For example, a latent 
variable of socioeconomic status may be made up of the observed variables of education, income, and professional status. 
Our analysis proceeded in a two-step fashion common when using SEM . First, we combined observed variables (i.e., the 
scale scores from the teacher surveys) to create one latent variable of G eneral Data U se. Second, we tested the proposed 
relationships between the latent variable of data use and student achievement. All SEM s were conducted within a multi-level 
framework that accounted for the nested nature of the data. 

H ierarchical L inear M odeling (H L M ) 

Next, we specified a series of HLM analyses that examined the unique relationship between each key dimension of data use 
and student achievement. For these analyses, wecombined Working with Data and Instructional Responses into one scale. 
Although the theory of action assumes these are distinct aspects of data-use practice, our review of the bivariate correla- 
tions among these two scales showed that they were highly correlated for both teachers and principals. We re-named the 
combined variable "Attention to Data in the Classroom" for teachers and "Attention to Data in the School" for principals. 

Therefore, the four independent variables we tested in separate HLM s were (1) Context, (2) Supports for Data Use, (3) At- 
tention to Data in the Classroom/School, and (4) Barriers to Data Use. 

HLMs were used to adjust for the nesting of students within teachers within schools. HLM models allow for simultane- 
ous estimates at level 1 (students), level 2 (teachers), and level 3 (schools). This modeling approach yields more accurate 
standard errors and regression estimates and also allows for including covariates at the student, teacher, and school levels 
(Raudenbush & Bryk, 2002). 

As with the other analyses including the SEM s, we analyzed separate HLM s by grade level (elementary and middle) and 
subject (reading and mathematics). We began modeling by including all student-, teacher-, and school-level covariates that 
were significantly related to student achievement (for a list of all possible covariates, see Appendix D, Table D.4). After 
the control model was developed, we then assessed the relationships between each key dimension of data use and student 
achievement, controlling forthecovariates in multiple sets of three-level models. Centering was used to make the estimates 
more interpretable (see Appendix D). 

Research Question 1: What Are the Relationships Between Teachers' Data-Use Practices 
and Perceptions and Their Students' Achievement? 

Teachers' General Data Use and Student Achievement 

We used multi-level SEM to test the hypothesized relationships between teachers' General Data Use and student achieve- 
ment. These were a series of two-level models with students nested withi n teachers, one each for elementary grades readi ng, 
elementary grades mathematics, middle grades reading, and middle grades mathematics. 


Council of the Great City Schools • American Institutes for Research • Summer 2012 


RESULTS CONT’D 


Our first step was to use the scale scores for each key dimension to create one large latent variable of General Data Use 
For this, we used the original four scales measuring the key dimensions Context, Supports, Vtorking with Data, and Instruc- 
tional Responses All four loaded well on the latent variable of General Data Use with factor loadings greater than 0.40, 8 
consistent with the finding that the dimensions of data use all were related, and further suggesting that they measured the 
one underlying construct of General Data Use (see Table 3.13). 

Table 3.13. Factor Loadings for the Latent Variable of Teachers' 

General Data Use in Each Analytic Sample 


T eacher Data- Use Scales 

M athematics 

E lementary M iddle 

Reading 
E lementary 

Middle 

Context 

1.00 

1.00 

1.00 

1.00 

Supports for data use 

0.72 

1.51 

2.28 

1.30 

Working with data 

0.54 

1.31 

3.05 

1.74 

Instructional responses 

0.72 

1.58 

3.12 

2.15 


Note. The factor loadings for Context were set to be 1.00 in the SE M models to set the metric for the other 
indicators in the latent variable. 


We next added in the path between teachers' General Data Use and student achievement, as measured on the state assess- 
ments in mathematics and reading, as depicted in Exhibit 3.1. 

Exhibit 3.1. Structural Equation Model of the Relationship Between 
Teachers' General Data Useand Student Achievement 





8 A cutoff of 0.40 is the standard acceptable level for factor loadings (Kline, 2005). 
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Table 3.14 reports the coefficients of the path between teachers’ G eneral D ata U se and student achievement for each grade 
level and each subject (represented by the question mark in Exhibit 3.1). Full results, including all paths and covariances 
for each model, are presented in Tables D.1-D.3 in Appendix D. 9 

Table 3.14. Relationships Between Teachers' General Data Useand 
Student Achievement in Elementary and Middle Grades Mathematics and Reading 


G rade L evel and C ontent A rea 

C oefficient 

Elementary grades mathematics 

0.04 

M iddle grades mathematics 

0.10* 

Elementary grades reading 

0.17* 

M iddle grades reading 

0.06 

Note. *p < .05, two-tailed. 


These results suggest that teachers' General Data-U se practices and perceptions are positively related to student achieve- 
ment in elementary grades reading and middle grades mathematics. However, there was no statistically significant relation- 
ship between teacher data use and student achievement in elementary grades mathematics and middle grades reading. These 
findings in combination partially support the theory of action. 

Links Between Each Key Dimension of Teacher Data Useand Student Achievement 

To further address the first research question, we estimated the unique associations among each teacher data-use dimension 
and student achievement using HLM and the methods described previously. These analyses controlled for student back- 
ground characteristics and prior achievement and teacher demographics. 10 

The results are shown in Table 3.15. They show that teacher reports of Context and Supports for Data Use were not sig- 
nificantly associated with student achievement in mathematics in either elementary or middle grades. Teachers’ Attention 
to Data in the Classroom was positively associated with mathematics achievement among middle grades students but not 
elementary grades students. Barriers were significantly and negatively correlated with elementary grades students’ math- 
ematics achievement but not middle grades students. 

These results show that in the middle grades, teachers' Attention to Data in the Classroom was correlated with higher stu- 
dent achievement in mathematics. In the elementary grades, teachers' perceived Barriers to data use were negatively related 
with student achievement such that the greater the perceived barriers to using data, the lower the students' achievement. 

D ifferent patterns emerged for reading achievement. A s shown in T able 3.15, for the elementary and middle grades samples, 
Context, Supports for Data U se, and Barriers were not significantly related to reading achievement. Attention to Data in 
the Classroom was positively associated with elementary grades student achievement in reading but notin middle grades. 


9 Model fit for both measurement and path models was highly similar and indicated moderate model fit. Comparative Fit Index (CFI) and Tucker Lewis Index (TLI) ranged 
between 0.80 and 0.95, which are less than the recommended value of 0.95. Root M ean Square Error of Approximation (RM SEA) of 0.08 is borderline, and Standardized 
Root Mean Square Residual (SRMR) of 0.09 is greater than the recommended value of 0.05. However, the statistically significant R2 value of 0.83 for both the reading 
and mathematics analyses indicates that the models account for a substantial percentage of variance in the outcome (83 percent). 

10 It is interesting to note that when conducting simple bivariate correlations between the data-use variables and student achievement scores, contrary to the hypothesized 
positive relationships between data use and student achievement, many of these correlations were negative in direction, suggesting that teachers with higher scores on the 
data-use scales in 2009-10 taught students with lower achievement in the prior spring (2009) and in the spring of 2010. These bivariate correlations do not control for 
any other variables, such as student prior achievement or socioeconomic status, nor do they adjust for the nested nature of the data. However, these negative relationships 
were later shown to be inconsistent with the results of the main analyses that did control for all relevant student, teacher, and principal background characteristics and 
were conducted in a multi-level framework. 
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Table 3.15. Relationships Between Teacher Data- Use Scales 
and Student Achievement in M athematics and Reading 


T eacher Data-Use Scales 

M athematics 
Elementary Middle 

Reading 
E lementary 

M iddle 

Context 

0.03 

0.04 

0.03 

0.06 

Supports for data use 

0.01 

-0.01 

-0.01 

0.01 

Attention to data in classroom 

0.04 

0.09** 

0.06* 

0.02 

Barriers to data use 

-0.08* 

-0.04 

-0.02 

0.00 


Note. *p < .05, **p < .01, two-tailed. 


Research Question 2: What Are the Relationships Between Schools' Policies, Practices, 
and Resources for Data-Driven Instruction and Student Achievement? 


Principals' General Data Use and Student Achievement 

To test the second research question, we drew on the principal surveys to measure school-level data-use practices and per- 
ceptions. Similar to the teacher-level analyses, we examined school-level general data use using multilevel SEM 11 and then 
examined the unique links between each dimension of school-level data use and student achievement with HLM . 

The process of creating a measure of principals' General Data-Use practices mirrored that described previously for teach- 
ers. The principals' scale scores for each of the four original key dimensions (Context, Supports for Data Use, Working with 
Data, and Instructional Responses) were used to build one latent variable. As for teachers, the four indicators loaded well 
on the latent variable of principals' General Data Use with all loadings greater than the 0.40 cutoff, which suggested that 
the four key dimensions of principal data use were related and measured the same underlying construct of General Data 
Use(seeTable 3.16). 

Table 3.16. Factor Loadings for the Latent Variable of Principals' 

General Data Usein E ach A nalytic Sample 


Principal Data-Use Scales 

M athematics 
Elementary Middle 

Reading 
E lementary 

Middle 

Context 

1.00 

1.00 

1.00 

1.00 

Supports for data use 

1.48 

1.62 

1.46 

1.75 

Working with data 

1.75 

4.84 

2.79 

1.69 

Instructional responses to data 

1.83 

5.73 

2.85 

1.95 


Note. The factor loadings for Context were set to be 1.00 in the SE M models to set the metric for the other 
indicators in the latent variable. 


We next added in the path between principals' General DataUseand student achievement. Therefore, the four school-level 
SEM models included one latent variable of principal data use and one observed variable of student achievement, as de- 
picted in Exhibit 3.2. 


11 The SEM s for the second research question were two-level models with students nested within schools. 
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Exhibit 3.2 Structural Equation M odel of Principals' 
General Data Useand Student Achievement 





Table 3.17 reports the coefficients of the path between principals’ General Data Useand student achievement for each grade 
level and subject (represented by the question mark in Exhibit 3.2). Full results, including all paths and covariances for each 
model, are presented in Tables D.1-D.3 in Appendix D. 12 

Table 3.17. Relationships Between Principals' General Data Useand Student Achievement 
in E lementary and M iddle G rades M athematics and Reading 


G rade L evel and C ontent Area 

Coefficient 

Elementary grades mathematics 

0.17 

M iddle grades mathematics 

0.23* 

Elementary grades reading 

-0.07 

M iddle grades reading 

0.01 

Note. *p < .05, two-tailed. 


The results suggest that principals' General Data-Use practices are positively related to student achievement in middle 
grades mathematics. We observed a statistically significant positive relationship between the latent variable of principals’ 
General Data Useand student achievement in only this one of the four analytic samples. 


12 Model fit for both measurement and path models in reading and mathematics were highly similar and indicated moderate model fit. CFI and TLI ranged between 0.7 
and 0.8, which are less than the recommended value of 0.95. RM SEA of 0.08 is borderline, and SRM R of 0.09 is greater than the recommended value of 0.05. However, 
the statistically significant R2 value of 0.83 for both the reading and mathematics analyses indicates that the models account for a substantial percentage of variance in 
the outcome (83 percent). 
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Links Between Each Key Dimension of Principal Data Use and Student Achievement 

For these analyses, we estimated a series of three-level models (level 1, students; level 2, teachers; and level 3, schools) to 
examine the unique links between each dimension and student achievement. No teacher data-use variables were included 
in these models; they included principal data-use variables only. All covariates included in the teacher models for the first 
research question were included in the principal models; these covariates included student background characteristics and 
prior achievement and teacher demographics. Principal covariates were tested, but none were significant predictors and they 
were therefore removed from the final models (see Appendix D, Table D.4). 13 

As shown in Table 3.18, principal-reported Supports for Data Use and Attention to Data inthe School were positively cor- 
related with student achievement in mathematics in elementary school. In addition, higher levels of Supports for Data Use 
reported by principals of elementary schools were associated with higher levels of student achievement in reading. Context 
and Barriers as reported by principals in elementary schools were not significantly correlated with student achievement in 
either subject. 


Among middle school principals, no significant relationships between the data-use variables and student achievement were 
observed in either mathematics or reading. 

Table 3.18. Relationships Between Principal Data-Use Scales 
and Student Achievement in M athematics and Reading 


Principal Data-Use Scales 

M athematics 
E lementary M iddle 

Reading 
E lementary 

Middle 

Context 

0.09 

0.03 

0.04 

-0.06 

Supports for data use 

0.11* 

0.06 

0.09* 

0.01 

Attention to data in the school 

0.10* 

0.04 

0.01 

0.00 

Barriers to data use 

-0.03 

-0.02 

-0.04 

0.10 


Note. *p < .05, two-tailed. 


13 As with the teacher analyses, bivariate correlations that did not control for any covariates, nor adjust for the nested nature of the data revealed negative and significant 
correlations between principal data use and student achievement, suggesting that schools with higher scores on the data-use scales in 2009-10 had students with lower 
achievement in the prior spring (2009) and in the spring of 2010. Again, these negative relationships were later shown to be inconsistent with the results of the main 
analyses that did control for all relevant student, teacher, and principal background characteristics and were conducted in a multi-level framework. 
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CHAPTER 4 

DISCUSSION AND 
CONCLUSION 


DISCUSSION AND CONCLUSION 


Although there was some prior evidence that using periodic assessments (formative assessments, progress monitoring, and/ 
or CBM ) may be positively related to student achievement, research on interim assessment use is limited. At the same time, 
significant investment has been made in interim assessment systems in school districts across the country. There is a great 
need for information about whether and how general and specific aspects of teachers’ and principals’ data-use practices and 
perceptions are linked to student achievement. This study attempted to fill this gap by measuring multiple aspects of interim 
assessment data use among teachers and principals, and empirically testing the links between key data-use practices and 
student achievement on end-of-year state assessments. 

Summary of Findings 

The study examined the relationships between student achievement and (1) teachers' practices and perceptions related 
to the use of interim assessment data and (2) school data policies, practices, and resources. We hypothesized that general 
and specific data-use practices and perceptions would be positively related to student achievement. The findings partially 
supported this hypothesis. For both teachers' and principals' General Data Use was related to student achievement in 
some grade levels and subjects. With regard to more specific practices and perceptions, teachers’ Attention to Data in the 
Classroom, principals' Attention to Data in the School, and principals' perceptions of Supports for Data Use were related 
to higher student achievement in some grades and subjects. In other words, the more that teachers and principals reported 
reviewing and analyzing student data and using this information to make instructional decisions, the higher their students' 
achievement, at least in some grades and subjects. M oreover for principals, the more they reported having support in the 
form of an appropriate data infrastructure, adequate time for review and discussion of data, professional development, and 
the appropriate human resources, the higher their students' achievement. Again, these results varied by grade and content 
area, with significant links observed in both elementary grades and middle grades, as well as in mathematics and reading. 
This section of the report considers the findings and their implications. 

Mean Differences in Data Use by Grade 

Average levels of reported data use and associated data-use-related perceptions were higher in the elementary grades than 
in the middle grades among both teachers and principals. This difference by grade level also persisted across content ar- 
eas, where both mathematics and reading teachers reported higher use in the elementary grades than in the middle grades. 
Specifically, elementary grades teachers reported more positive perceptions of the data culture and supports for data use in 
their schools and school districts and also reported spending more time reviewing interim assessment data and using this 
information to make instructional decisions in the classroom than did middle grades teachers. These differences may be due 
in part to the fact that elementary grades teachers often teach all subjects, have fewer students in total, and interact with their 
students for more time during the school day than do middle grades teachers. This may promote a more supportive data cul- 
ture in which teachers and administrators are more likely to work with the data and engage in data-driven decision making. 
M ore support may be necessary to facilitate higher levels of data use in middle schools in urban districts. 

Cor relations Among Key Dimensions of Data Use 

The teacher and principal surveys measured the key dimensions of data use as depicted in the theory of action that guides 
this study (see Exhibit 1.1). As expected, there were positive correlations among all the key dimensions of data use and a 
negative correlation between each dimension and Barriers to Data Use. Most of the positive correlations were moderate to 
high, suggesting that these dimensions are indeed related as posited by the theory of action. Two of the key dimensions— 
Working with Data and Instructional Responses— were so highly correlated that, although we set out to measure them as 
separate aspects of data use, they were too interconnected to consider them distinct. 
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What Are the Relationships Between Teachers' Data-Use Practices and Perceptions and Their Students' 
Achievement? 

The hypothesized relationship between teachers' use of interim assessment data and student achievement was partially sup- 
ported through both of the main analysis strategies used in this study. 

Teachers' General Data Use and Student Achievement. Teachers' General Data Use (a combination of the four key 
dimensions in the theory of action) was related to student achievement in elementary grades reading and middle grades 
mathematics. These findings suggest that the overall interim assessment process — including the Context and data culture, 
concrete Supports for Data U se, and actual review and use of the data— may be a promising practice in urban districts in 
some grade levels and content areas. The magnitude of the relationships was modest, with effect sizes of 0.10 for middle 
grades mathematics and 0.17 for elementary reading. Shifting a student's test score by 0.17 standard deviations could have 
a significant effect on his or her academic standing. For example, if a student who was at the 50th percentile at the end of 
grade 3 had a fourth grade teacher who was at the mean on General Data Use, that student would be at the 57th percentile 
at the end of grade 4. This could be the difference between a student being categorized as below proficient and proficient on 
a state assessment. Also, if a student were in classrooms for multiple consecutive years with teachers who have strong data- 
use perceptions and practices, this positive advantage could be cumulative over time, possibly contributing substantially to 
the student's academic achievement. 

However, we did not find a significant relationship between teachers’ G enera I D ata U se and student achi evement i n el emen- 
tary grades mathematics or middle grades reading. We conducted district-specific analyses to test whether the nonsignificant 
relationships were the result of averaging district-specific effects that varied from each other (e.g., some positive and some 
negative). We found that the district-specific relationships in each subject and grade level appeared to be relatively similar in 
direction and magnitude. That is, in all four districts, there was no significant relationship between teachers’ G enera I D ata 
U se and student achievement in elementary school mathematics and middle school reading, and the district- average results 
reported in the Results section fairly represented each participating district. 

Teachers’ Specific Data-Use Practices and Perceptions and Student Achievement. The results of analyses testing the 
unique links between the key dimensions of data use and student achievement also partially supported the theory of ac- 
tion. Specifically, we found that teacher-reported Attention to Data in the Classroom was positively associated with student 
achievement in middle grades mathematics and elementary grades reading. That is, the more teachers reported reviewing 
interim student data and responding in the classroom, the higher their students' achievement on the end-of-year state assess- 
ment. 

These findings suggest that teachers’ review and response to interim assessment data can potentially act as a lever to im- 
prove student achievement in urban districts. Teachers' Attention to Data in the Classroom was more strongly related to 
student achievement than their sense of the assessment and instructional context, data culture, or supports for data use. This 
key finding is consistent with previous research that suggests that simply having interim assessments in place is not enough, 
and that to be effective, educators must actually use data to identify problems, identify reasons behind the problem, and then 
determine how to adjust their teaching to address the problems (Anderson etal., 2010). By linking teachers' data-use prac- 
tices and perceptions with their own students' achievement, this study extended prior research that suggested that working 
with data may help teachers understand and identify their students' needs (e.g., Quintet al., 2008). 
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Our results indicate that teachers’ review of data and subsequent instructional responses were the data-related practices and 
perceptions most strongly linked to improved student achievement and can be a focus for intervention and improvement 
with teachers in both elementary and middle grades. To consider in a practical sense how Attention to Data in the Classroom 
might be improved or addressed in urban schools, it is useful to break this construct down into its component parts, and 
further into the specific practices and activities that comprise the component parts. 

Attention to Data in the Classroom was a combination of teachers' Working with Data and their data- based Instructional Re- 
sponses. There were three subscales included in Attention to Data in the Classroom: teachers' individual attention to data, 
collaboration around data and thei r instructional responses to data Each of these was composed of items reflecting a 
number of specific practices related to teachers’ review of and response to student data. Examples of the specific practices 
are shown in Table 4. 1 . Any or all of the practices shown in Table 4. 1 may contribute to or drive the significant positive link 
between Attention to Data in the Classroom and student achievement that we found in this study. 
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Table 4.1. Components and Specific Practices that Comprise 

Attention to Data in the C lassroom 


Components of Teachers’ 
Attention to Data in the 
Classroom 

C oncrete and Specific Attention to Data Practices 

Individual Attention to Data 

Teachers’ independent review, analysis, and interpretation of 
their student data, such as: 

o 1 dentifying the number of students per proficiency 
category 

o Reviewing the percent of students who mastered each 
separate item or groups of items on the interim 
assessment 

o Identifying trends in content mastery at the individual 
student level and classroom level 
Includes frequency of review and overall amount of time 
spent engaging in independent review 

Collaboration around Data 

Teachers’ review, analysis, and interpretation of data in 
collaboration with other teachers, administrators, instructional 
coaches, data coaches, parents, and with students. 

Frequency of participation in formal “data meetings” or 

professional learning communities 

Frequency of participation in informal collaborative 

meetings/discussions 

Instructional Responses to 
Data 

On the basis of gaps and strengths identified in interim 
assessment data, instructional strategies such as: 

o Adjusting lesson plans (e.g., to spend more or less 
time on a concept than originally planned, depending 
on needs identified in the data) 
o Changing scope or sequence of instruction 
o Reteaching missed or misunderstood material or 
concepts to the whole class, small groups, or 
individual students 

o Changing teaching methodology (e.g., from lecture to 
activity- based) 

o Changing or adapting instructional materials 
o Re-grouping 

■ Heterogeneously, to mix students with 
different skill/mastery levels 

■ Homogeneously, to provide remediation or 
acceleration to students at similar skill levels 

o Providing targeted interventions for students with poor 
performance on interim assessments, such as referring 
students for tutoring within and outside of school. 
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The positive link between Attention to Data in the Classroom and student achievement indicates that the more that teach- 
ers engage in the types of practices listed in Table 4.1, the higher their students' achievement. At the component level, this 
means that the more that teachers engage in independent review of their data, and collaborate with others to review their 
students’ data, and the more they can point to specific instructional responses to data that they use with their students, the 
higher their students' achievement at the end of the year. 

Of course, this study was not designed to determine whether any or all of these data-use practices shown in Table 4.1 cause 
achievement to increase, but this study does provide evidence to suggest that supports that encourage these practices may 
hold promise for improving the use of interim assessment data, which in turn may help improve student achievement. That 
is, our results suggest that teachers' review and response to interim assessment data as described above can potentially act 
as a lever to improve student achievement in urban districts. 

Teachers' perceived Barriers to Data Use (such as a lack of time to study and think about data, a lack of time to collaborate 
with others in analyzing and interpreting data, or a lack of timeliness in receiving students' scores) were negatively related 
to student achievement— but only in elementary grades mathematics. As depicted in the theory of action, perceived barriers 
to using data can interrupt the assessment process at any point. For example, if teachers perceive that the test itself is of low 
quality, they will be unlikely to even examine thei r students' results closely. If teachers accept the validity and quality of 
the test but do not receive the data in an easy-to-use format, they still may not be able to review the data with ease and then 
respond to students' strengths and weaknesses within their classrooms. Perceived Barriers to Data Use may be indicative 
of breakdowns with the interim assessment process specifically or may more generally be an indicator of systemwide or 
districtwide issues. Additional investigation should examine why barriers have a more negative effect on elementary grades 
mathematics than the other subjects and grade levels. 

WhatArethe Relationships Between Schools' Policies, Practices, and Resources for Data-Driven Instruc- 
tion and StudentAchievement? 

As with the teacher- reported data-use practices and perceptions, the hypothesized relationships between schools' policies, 
practices, and resources for data-driven instruction and student achievement were partially supported. 

Principals' General Data Use and StudentAchievement. Principals' General Data Use (again, a combination of the 
four key dimensions of data use i n the theory of action) was related to student achievement in middle grades mathematics. 
However, there was no relationship in middle grades reading, elementary grades mathematics, or elementary grades read- 
ing. These findings suggest that the overall interim assessment process may be a promising practice at the school level in 
urban districts for some students but not all students and subjects. Principal-reported data-use practices and perceptions 
were related to improvement in student achievement only in middle grades mathematics. Thus, it may be possible that the 
adoption of an interim assessment process with an emphasis on supporting effective data use may be one area in which urban 
schools can intervene at the school level and positively impact student achievement, perhaps in middle grades mathematics 
in particular. 

Principals’ Specific Data-Use Practices and Student Achievement. The results of analyses testing the unique links 
between the school-level key dimensions of data use and student achievement also partially supported the theory of ac- 
tion. Principal-reported Supports for Data Use and Attention to Data in the School were positively associated with student 
achievement in some grades and subjects. 
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Principals' reported levels of working with and responding to interim assessment data (i.e., Attention to Data in the School) 
were positively related to elementary school students' mathematics scores. For principals, working with and responding to 
data were school-level processes, such as targeting teacher professional development on the basis of interim assessment 
results or adapting school improvement plans on the basis of interim assessment results. A Ithough this study cannot purport 
a causal link between these school-level processes and student achievement, this finding is consistent with previous research 
that suggests school-level processes around data use are promising for improving student achievement (Henke, 2005; U.S. 
Department of Education, 2010a). The magnitude of the link between principals Attention to Data and student achievement 
in elementary mathematics was educationally meaningful, with an effect size of 0.10. 

Again, to make this key finding actionable in terms of identifying specific practices on which schools and districts could 
focus when seeking to improve the use of interim assessment data, it is useful to break Attention to Data in the School down 
into its component parts and further, into the concrete activities that comprise the component parts. 

Attention to Data in the School was a combination of principals' review and analysis of data and their data-based responses. 
The three subscales included in Attention to Data inthe School were principals' individual attention to data collabora- 
tion around data and schooi-le/el responses to data Each of these was composed of items reflecting a number of specific 
practices related to principals’ review of and response to student data. Examples of the specific practices are shown in Table 
4.2. As with the classroom-level practices for teachers, any or all of the school-level practices shown in Table 4.2 may con- 
tribute to or drive the significant positive link between Attention to Data in the School and elementary math achievement 
that we found in this study. 
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Table 4.2. Components and Specific Practices that Comprise Attention to Data in the School 


C omponents of Principals’ 

Attention to Data in the 
Classroom 

Concrete and Specific Attention to Data Practices 

Individual Attention to Data 

Principals’ independent review, analysis, and 
interpretation of their student data, such as: 

o Identifying the percentage of students scoring at 
or above the proficiency level by grade, subject, 
and classroom 

o Comparing the performance of students in their 
school versus other schools 
o Examining the performance of student subgroups 
(e.g., students with disabilities, English learners) 
o Identifying changes or trends in the school’s 

results across years 

Includes frequency of review and overall amount of time 
spent independently reviewing data 

Collaboration around Data 

Principals’ review, analysis, and interpretation of data in 
collaboration with other teachers, administrators, 
instructional coaches, data coaches, parents, and with 
students. 

Frequency of participation in formal “data meetings” and 

professional learning communities 

Frequency of participation in informal collaborative 

meetings/discussions 

Instructional Responses to Data 

On the basis of data review: 

o M aking curriculum changes or decisions 
o Developing school improvement plans 
o Seeking professional development for teachers 
based on identified gaps in either content or 
pedagogical skills that are revealed in the data 
o Setting schoolwide student achievement goals 
o Evaluating programs (i.e., examining trends over 
time for students who participate in particular 
instructional programs/ initiatives) 


Given the positive relationship between Attention to Data in the School and student achievement, the more principals 
engage in the types of activities shown in Table 4.2, the higher the math achievement of students in their schools (at the 
elementary level). Specifically, independent review of interim assessment data may represent a set of promising principal or 
school-level data-use practices. M oreover, collaboration that includes administrators— either principals or assistant princi- 
pals, or both where appropriate— may be a key feature of an effective data-use process in urban schools. If supported, this 
type of collaboration may potentially help drive improvements in student achievement. Finally, the specific examples of 
school-level responses shown in Table 4.2 may be promising data-use activities. 
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Our final key finding was that principals’ perceptions of Supports for Data Use were also positively related to student 
achievement in both elementary grades reading and mathematics. The literature on school-level data use emphasizes the 
importance of organizational supports— such as common meeting times to discuss data, the presence of a data coach, the 
quality of the data infrastructure, and professional development around data use— as promising dimensions of effective data 
use (Bulkley etal., 2010; Clune& White, 2008; Henke, 2005; M arsh etal., 2006; Young, 2006). Our results further indicate 
that these are aspects of using data most strongly related to student achievement at the principal/school level, with effect 
sizes ranging from 0.09 to 0.11. 

As with teachers' Attention to Data in the Classroom and principals' Attention to Data in the School, it is useful to break 
down Supports for D ata U se into its component parts and the concrete and specific aspects of these components in order to 
consider ways that schools and school districts can improve their data supports to be more effective. 

The three subscales included in Supports for Data Use were organizational supports; staffing and human resources, 
and data infrastructure In general, the more positive principals' perceptions of these supports, the higher their students' 
achievement (at the elementary school level). Table 4.3 shows the specific aspects of each of these three components. Any 
or all of the aspects shown in Table 4.3 may contribute or drive the significant positive link between Supports for D ata U se 
and student achievement that we found in this study. 

Any or all of these specific aspects in Table 4.3 may be key school-level supports that helped drive the link between princi- 
pals' perceptions Of their Supports for Data U se and student achievement. Further research is needed to examine the finer 
grained relationships, but the results of this study suggest that the concrete and specific Supports descri bed above are poten- 
tially promising aspects of interim assessment data use. 

The observed findings for principals were in contrast to the results focused on teachers, where Supports for D ata U se were 
not a significant predictor of student achievement at either grade level or subject. Principal-reported Supports for D ata U se 
included both the support that they themselves provide in an administrative role as well as supports principals and teach- 
ers receive from the school district. Our findings suggest that these school and district supports hold promise as levers for 
change in urban schools to improve student achievement in elementary grades reading and mathematics. 

However, contrary to our hypotheses, principals’ data-use practices and perceptions were not significantly correlated with 
middle grades students' achievement in either mathematics or reading. These differences between elementary and middle 
grades should be examined in future research. 
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Table 4.3. Components and Specific Aspects of Supports for Data Use 


C omponents of Supports for 
Data Use 

Concrete and Specific Supports for Data Use 

Organizational Supports 

Structured time for review and discussion of interim 
assessment results built into the school day for teachers 
and administrators 

o Sometimes conducted with the whole staff, 
subject area teams or departments, or at grade 
level data meetings 

Data coaches who conducted such activities as: 

o Providing feedback on school improvement plans 
that incorporate student achievement data 
o M aking recommendations about curricular or 
instructional changes based on student scores 
o Emphasizing the link between instructional 
practices and student interim assessment scores 

Staffing and Human Resources 

Principals’ perceptions of the quantity and quality of the 
professional development offered to their teachers that is 
specifically about using data to inform instruction 
T raining on how to access student data electronically, 
how to generate different types of reports, and how to 
analyze and respond to student data . 4 
Staff capacity to use data including principal perceptions 
of the ability of their teachers to use data in multiple 
ways, such as 

o T ranslating data into knowledge about student 
strengths and weaknesses 
o Analyzing trends in individual student- and 
classroom- level performance overtime 
o M aking data-based instructional changes 

Data Infrastructure 

Quality, timeliness, and ease of use of the data system 
including: 

o The ways that principals access student interim 
assessment data (e.g., electronically or on paper- 
based reports) 

o Lag-time to gain access to student data after 
administration of assessments 


14 It is important to note that both principals and teachers indicated needing more support and training in how best to respond to student data. 
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Study Limitations 


This study used rigorous statistical modeling to explore the relationships between the key dimensions of data use and student 
achievement, but it is important to note some limitations. First, the study relied on self-report survey data rather than obser- 
vations of actual data use and instructional practices. Based on the teacher and principal survey results, the four key dimen- 
sions of data use were highly correlated. A Ithough these correlations provide some basic support for the study's theory of 
action, they may be at least partially explained by measurement error due to the fact that the variables were measured using 
the same survey. If the key dimensions were measured separately with different techniques (e.g., a combination of survey 
and observations), we may have been able to obtain more refined measures of each dimension and the key elements and 
components within each dimension. These constructs in the theory of action provided a roadmap for designing the surveys 
used for this study; however, as measured, the scales (and subscales) derived from the surveys did not appear to represent 
highly distinct aspects of data-use practices and perceptions. Future studies may benefit from a mixed-methods approach to 
measuring teacher and principal data use. 

A related limitation is that the analysis does not identify whether certain practices within these dimensions are more prom- 
ising than others. Future work also should continue to refine the measurement strategy to allow for analyses of the links 
between more specific data-use practices and student achievement than could be tested in this foundational study. 

A third limitation of the study is that it was not designed to provide information about the implementation quality of the 
interim assessment process in the participating districts. For example, a teacher may report regularly attending team data 
meetings, but the quality or relevance of the content of those meetings is not captured in the survey responses. The survey 
data also do not provide information about the quality of the actual interim assessments used in the participating districts. 
Although we collected data about perceived alignment with the curriculum, pacing guides, and state assessments, a measure 
of the true degree of this alignment was beyond the scope of this study. To partially address this limitation, we conducted a 
follow-up study of the alignment between the interim assessments and the state standards and pacing guides in one subject 
(mathematics) in one of the participating districts. A brief report on this alignment study is provided in Appendix A. The re- 
sults suggest that in this one district, the interim assessments were well aligned with the state assessment. Similar alignment 
studies in the other participating districts would help to further ground the results of the main study. 

A fourth limitation involves the generalizability of the findings. The school-based samples of principals, teachers, and stu- 
dents were sufficiently large in size, but there were only four districts. It is not clear whether the findings from these four 
districts can be generalized to other urban districts or districts in other localities (e.g., suburban or rural). 

Finally, although this study provides evidence of a relationship between some key dimensions of data use and student 
achievement, it is essential to understand that no causal claims about the nature of these relationships can be made on the 
basis of this correlational study. Interim assessments and the use of their data arejustoneof a number of policies, practices, 
and interventions being implemented within schools and school districts. It is not our claim that supports for data use and 
attention to data use in the classroom or school causeimproved student achievement in certain grades and subjects. Rather, 
this study provides foundational evidence that as some aspects of data use increase, so too does student achievement. This 
study does not rule out the possibility that something else caused both the level or degree of data use and the improvement 
in achievement. 
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Future Directions 

Future research can help provide additional evidence of whether and how interim assessments can be used as a tool to in- 
crease student achievement. The ultimate goal would be to develop a set of standards or strategies that districts and schools 
can use as a guide for effective data use. 

Self-report measures of data use have provided valuable information on how interim assessments are used at classroom and 
school levels in the selected urban districts. As already noted, these data could be further enhanced through observations 
of actual data-use practices. These could include observations of data meetings where staff discuss the results of interim 
assessments or plan instructional responses and professional development on data use. Deeper study could also include a 
review of lesson plans that stem from a review of the interim assessment data, along with classroom observations to explore 
how these plans are implemented. 

As we learn more about specific data-use practices that are associated with student achievement, another next step is to de- 
velop a more refined theory of action and set of measures designed to tap into it. This could involve the suggestions already 
mentioned, followed by developing specific data-use interventions intended to improve teachers’ instructional responses 
and student achievement that can be systematically tested in the schools. 

Of the six significant relationships between data-use practices/perceptions and student achievement that emerged in this 
study’s analyses, five were found in the elementary grades. This finding suggests that something about the structure of the 
elementary grades may be more conducive to the successful use of interim assessments, compared with the middle grades. 
Further research is needed to identify key factors in the elementary grades that could be adopted or adapted in the middle 
grades to potentially increase the utility of interim assessments for older students. Similarly, further exploration can help 
achieve a better understanding of data-use differences in mathematics and reading that were revealed in this study. 

Conclusion 

With the current increase in the use of interim assessments, the need for a more comprehensive body of literature on ef- 
fective use of data for instructional improvement is critical. Supporters of interim assessments believe that using this type 
of measure on a periodic basis can lead to improved student achievement. Flowever, despite the widespread use of these 
assessments, few studies actually document the relationship between data-use-related perceptions and practices and student 
achievement. This study attempted to shed light on this issue by examining the relationship between key dimensions of data 
use and student achievement in two major content areas (reading and mathematics) and in the two grade levels (elementary 
and middle) in which interim assessments most often are used. The results across content areas and grade levels were mixed 
but suggest that some aspects of classroom- and school-level interim assessment data use are related to improvements in 
student achievement. The results also appear to be in line with previous research that suggests that having interim assess- 
ments may be helpful but not sufficient to produce positive changes in student achievement. 

This study sought to begin to understand the connection between different aspects of interim assessment data use and stu- 
dent achievement. Given that school districts and schools are facing significant budget challenges and must make important 
decisions about resource allocation, it is imperative that we identify the specific dimensions of data use that are most im- 
portant for improving student outcomes. M any school districts are increasingly using various types of assessments and data 
in an effort to engage in data-driven decision making. Although the study focused primarily on the use of data from interim 
assessments, some of the study results may extend beyond interim assessments to provide a glimpse into the overall data 
culture of participating districts. As such, this study provides a foundation for the future exploration of the relationships 
between student achievement and other types of data that can be used for instructional and school improvement. 
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Finally, these findings have implications for data-use policies and practices in school districts and schools. The findings 
suggest that, at the very least, if schools adopt interim assessments to produce changes in student achievement, schools and 
districts should provide adequate support for using the data, and teachers should actively use data in the classroom— both 
by spending time individually and col laborati vely reviewing the student data and by responding instructional ly. 

This is particularly important as the nation moves toward Common Core State Standards and the assessment systems that 
will accompany them. Although these findings do not identify the specific aspects of each dimension that are most impor- 
tant, it appears that data use by principals, particularly in elementary school, may be as important as teacher data use. This 
is in line with the findings from our site visits (as well as prevailing wisdom) that suggest that leadership and support from 
the administration are critical. The findings of this three-year project revealed that schools are better able to work with data 
when they have the appropriate data infrastructure, organizational supports for the analysis and productive discussions about 
data, human resources (e.g., data coaches) and professional development. In addition, there are important uses for interim 
assessment data by stakeholders at all levels. These include use by district leaders to identify professional development 
needs and evaluate district initiatives, use by school leaders to develop and evaluate school and staff improvement plans, and 
perhaps most importantly, use by teachers to inform instructional strategies. It is there in the classroom that student needs 
are met most effectively. 
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Appendix A: District and StateContext 

State Data-Use Capacity: America COM PETES Act 

The states of three of the participating districts have previously submitted Race to the Top grants that, as of fall 2011, have 
not been funded. I n their Race to the Top grant applications, the states described their plans to adopt formative and summa- 
tive assessments, monitor student growth, improve data systems, and make data more widely available. These states also 
described their status regarding the 12 America COM PETES Act elements (see Table A .1). Information on the status of data 
systems in the fourth district, in Virginia, was collected through the Virginia Statewide Longitudinal Data Systems (SLDS) 
grant application, which was submitted in 2010. Three of the four participating districts have in place all or nearly all the 
required elements at the state level, with plans to improve on them and implement the remaining elements. 

Table A. 1. Data Elements in the Four Participating Districts 


District 

A m erica C 0 M PE T E S Act E lements 

1 

2 

3 

4 

1. Unique student ID 

P 

c 

c 

c 

2. Student-level enrollment, demographics, and 
participation 

C 

c 

c 

c 

3. Student-level information about points at which a 
student exits, transfers in/out, drops out, or completes 

PK-16 

P 

c 

c 

c 

4. Capacity to communicate with higher education system 

C 

c 

c 

p 

5. State data audit assessing data quality, validity, and 
reliability 

c 

c 

c 

p 

6. Y early state assessment records of individual students 

c 

c 

c 

c 

7. Information on students not tested by grade and subject 

c 

c 

c 

c 

8. T eacher identifier system with ability to match 
individual teachers to individual students 

c 

c 

c 

— 

9. Student-level transcript information including course 
completion and grade earned 

c 

c 

c 

— 

10. Student-level college readiness test scores 

c 


c 

c 

11. Data on student transition from secondary to post- 
secondary, including remedial coursework enrollment 

c 

c 

c 

p 

12. Data necessary to address alignment and adequate 
preparation for success in post-secondary 

c 

c 

c 

p 


Note. Elements marked with a C are complete or operational but undergoing improvements. Elements marked with a 
P are partially completed. 
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District 3: In-Depth Site Visit Evaluation Examining Alignment of the Interim Assessment 
Process W ith the State Standards 

In J uly 2011, an independent third-party analyzed the degree to which the interim assessments in District 3 in grades 5 and 8 
were aligned with the state Core Content for M athematics expectations as delineated by the State Department of Education 
and the district pacing guides. All 143 items from the grade 5 and grade 8 interim assessments were analyzed individually to 
reveal if they were appropriately aligned to a state expectation and the instructional units in the pacing guide. The analysis 
revealed an extremely high degree of alignment between the grades 5 and 8 interim assessments and both the state's Core 
Content in M athematics and the county's pacing guide (see Table A .2). A I though data use can be impeded by the perception 
that assessments are not aligned to pacing guides and curricula, the independent reviewer concluded that the i nteri m assess- 
ments are connected with both the curriculum expectations and unit pacing. 


TableA.2. Summary of Alignment Study Findings 



G rade 5 

G rade 8 


n 

% 

n 

% 

N umber of proficiency assessments analyzed 

3 

- 

8 

- 

T otal number of items analyzed 

39 

- 

104 

- 


Items appropriately aligned with the core context expectations 

38 

97% 

102 

98% 

Items appropriately placed within pacing guide 

39 

100% 

101 

97% 

Items aligned with current unit(s) 

33 

85% 

69 

68% 

Items aligned with prior unit 

6 

15% 

32 

32% 


Council of the Great City Schools • American Institutes for Research • Summer 2012 


APPENDICES CONT’D 


Appendix B: Measures and Data-Use Survey Items 

This appendix includes the reliability statistics of each subscale and scale used to measure the key dimensions of data use 
for teachers and principals. A list of example data-use survey items administered to teachers and principals also is provided. 

• Tables B. 1 and B.2 present the reliability of each teacher subscale and scale (Table B. 1) 
and each principal subscale and scale (Table B .2). 

• Table B.3 presents example survey items from the teacher and principal surveys of data use. 

Teacher and Principal Data-Use Survey Reliability 

The reliability statistics of each scale and the number of survey items in each scale are shown in Table B.l for the four 
samples of teachers, including elementary grades reading teachers, elementary grades mathematics teachers, middle grades 
reading teachers, and middle grades mathematics teachers. The alpha statistics for all key dimensions of the data-use scales 
and subscales measuring teachers' data use are greater than 0.70, which suggests adequate reliability. The number of survey 
items that measure the key dimensions of data use for teachers range from 15 to 48 survey items. For example, among el- 
ementary grades reading teachers, the Supports for Data Use scale is internally consistent with a Cronbach's alpha of 0.95 
and is measured with 48 teacher survey items. 

Table B.2 provides reliability coefficients (Cronbach’s alpha) for scales and subscales measuring the key dimensions of data 
use for principals and the number of survey items that measure each subscale and scale. Each principal scale and subscale 
has a Cronbach's alpha greater than 0.70, which suggests that each scale and subscale is internally consistent. The number 
of survey items from the principal data-use survey used to measure the key dimensions of data use ranged from 13 to 51. For 
instance, the principal Supports for Data Use scale is internally consistent with a Cronbach's alpha of 0.92 and is measured 
with 51 survey items. 
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Table B.l. Reliability of Data-Use Scales and Subscales (Teacher Survey) 





Cronbach's Alpha 


ni^nancinn Subscales W ithin E ach K ey 
Sr Dimension Scale 

Survey 

Item/V 

Elementary 
Grades Reading 
T eachers 

Elementary Grades 
M ath T eachers 

Middle Grades 
Reading Teachers 

Middle Grades 
M ath T eachers 

C ontext 

36 

0.86 

0.86 

0.91 

0.88 

Assessment/Instructional Context 

23 

0.79 

0.78 

0.86 

0.81 

State, District and School Data 
Culture 

13 

0.85 

0.86 

0.89 

0.88 

Supports for Data Use 

48 

0.95 

0.95 

0.95 

0.94 

Data Infrastructure 

9 

0.80 

0.80 

0.79 

0.81 

Organizational Supports 

31 

0.95 

0.94 

0.94 

0.94 

Staffing/Human Resources 

8 

0.96 

0.95 

0.95 

0.94 

Working with Data 

42 

0.94 

0.94 

0.95 

0.95 

Individual Teacher Attention to 
Data 

16 

0.89 

0.88 

0.91 

0.90 

Teacher Collaboration A round 
Data 

10 

0.87 

0.88 

0.89 

0.88 

Teacher-Principal Collaboration 

3 

0.75 

0.75 

0.77 

0.67 

Teacher-Coach Collaboration 

4 

0.78 

0.79 

0.79 

0.77 

Teacher-Parent Collaboration 

4 

0.74 

0.74 

0.77 

0.74 

T eacher-Student Col laboration 

5 

0.89 

0.89 

0.89 

0.88 

Instructional Responses 

26 

0.96 

0.97 

0.97 

0.97 

Establish/ Adjust Groupings 

6 

0.84 

0.85 

0.86 

0.86 

Change Scope and Sequence 

5 

0.84 

0.84 

0.89 

0.87 

Adjust Lesson Plans 

10 

0.93 

0.93 

0.95 

0.94 

Provide Supplemental Resources to 
T argeted Students 

5 

0.82 

0.82 

0.85 

0.84 

Barriers to Data Use 

15 

0.74 

0.74 

0.79 

0.76 


Sample Size: E lementary Reading T eachers N = 612; E lementary M athematics T eachers N = 587; M iddle School Reading T eachers N = 552; M iddle School 
M athematics T eachers N = 473 
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TableB.2. Reliability of Data-Use Scales and Subscales (Principal Survey) 





Cronbach's Alpha 


Key Dimension Subscales Within Each Key 

Scale Dimension Scale 

Survey 
Item A 

General 

Principal 

Responses 

Reading 

Instruction 

Specific 

Math 

Instruction 

Specific 

Context 

32 

» 

0.87 

0.87 

Assessment/Instmctional Context 

23 

- 

0.82 

0.81 

State, District and School Data 
Culture 

9 

0.84 

- 

- 

Supports for Data Use 

51 

0.92 

— 

— 

Data Infrastructure 

8 

0.82 

- 

-- 

Organizational Supports 

31 

0.90 

- 

- 

Staffing/Human Resources 

12 

0.93 

-- 

-- 

Working with Data 

42 

— 

0.97 

0.97 

Individual Principal Attention to 
Data 

17 


0.93 

0.93 

Principal Collaboration Around 
Data 

3 

0.79 



Principal-T eacher Collaboration 

9 

0.86 

- 

- 

Principal-Coach Collaboration 

4 

0.83 

- 

- 

Principal-Parent Collaboration 

3 

- 

0.75 

0.75 

Principal-Student Collaboration 

6 

0.87 

- 

- 

Instructional Responses 

24 

— 

0.96 

0.97 

Establish/Adjust Groupings 

4 

- 

0.80 

0.80 

Adjust Lesson Plans 

6 

- 

0.88 

0.88 

Provide Supplemental Resources 
to T argeted Students 

3 


0.81 

0.81 

School-Level Instructional 
Response 

11 

- 

0.92 

0.93 

Barriers to Data Use 

13 

0.76 

- 

- 


Note. - denotes that the reliability coefficient was not calculated because the items included in the scale or subscale were either content specific (generating Reading 
Instruction Specific and/or Mathematics Instruction Specific coefficients, but not general principal responses) or were content general (generating a coefficient for general 
principal responses, but not content specific reading or mathematics coefficients). School-Level Instructional Response is a subscale specific to the principal survey. 
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Data -Use Survey Items 

Table B .3 presents a series of example survey items and response options from the data-use surveys distributed to teachers 
and principals. The teacher survey was administered to teachers of reading or mathematics in grades 4, 5, 7, and 8, and the 
principal survey was administered to principals and, where appropriate, assistant principals of participating elementary and 
middle schools. The example survey items are linked with the subscale and scale it measured. Each scale measured a key 
dimension of data use as depicted in the theory of action, including Context Supports for Data Use Working with Data, 
Instructional Responses, and Barriers to data use. 
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Table B.3. Example I terns from theTeacher and Principal Surveys of Data Use 


Scale 

Subscale 

Survey Item 

Response Scale 

C ontext 

Assessment/Instmctional 
C ontext 

How much do you agree or disagree with the following statement? 
The district interim assessments are well aligned with state and 
district standards. 

1 = Strongly disagree; 

2 = Disagree; 3 = Agree; 
4 = Strongly agree 

C ontext 

State, District, and 
School Data Culture 

How much do you agree or disagree with the following statement 
about your district’s priorities about using data? The district sets 
clear, consistent goals for schools to use data for school 
improvement. 

1 = Strongly disagree; 

2 = Disagree; 3 = Agree; 
4 = Strongly agree 

Supports for Data Use 

Data Infrastructure 

How much do you agree or disagree with the following statement 
about using district interim assessment data? Interim assessment 
results are reported to me in a timely manner. 

1 = Strongly disagree; 

2 = Disagree; 3 = Agree; 
4 = Strongly agree 

Supports for Data Use 

Data Infrastructure 

How much do you agree or disagree with the following statement 
about using district interim assessment data? Interim assessment 
data are easy to use. 

1 = Strongly disagree; 

2 = Disagree; 3 = Agree; 
4 = Strongly agree 

Working With Data 

Teacher Collaboration 
Around Data 

How frequently do you review student interim assessment data with 
classroom teachers? 

0 = N ever; 

1 = 1 or 2 times a quarter; 

2 = 1 or 2 times a month; 

3 = 1 or 2 times a week 

Working With Data 

Teacher-Coach 

Collaboration 

How frequently do you review student interim assessment data with 
instructional coaches? 

0 = N ever; 

1 = 1 or 2 times a quarter; 

2 = 1 or 2 times a month; 

3 = 1 or 2 times a week 

Working With Data 

Teacher- Parent 
Collaboration 

How frequently do you review student interim assessment data with 
parents/guardians? 

0 = N ever; 

1 = 1 or 2 times a quarter; 

2 = 1 or 2 times a month; 

3 = 1 or 2 times a week 

Working With Data 

Teacher-Student 

Collaboration 

How frequently do you review student interim assessment data with 
students? 

0 = N ever; 

1 = 1 or 2 times a quarter; 

2 = 1 or 2 times a month; 

3 = 1 or 2 times a week 

Working With Data 

Individual Teacher 
Attention to Data 

How much have you used the latest interim assessment results to 
identify individual students who need remedial assistance? 

0 = Did not use in this way; 

1 = Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Working With Data 

Individual Teacher 
Attention to Data 

How much have you used the latest interim assessment results to 
identify areas where you need to strengthen your content knowledge 
or teaching skills? 

0 = Did not use in this way; 

1 =Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Instructional Responses 

Adjust Lesson Plans 

How much have you used the latest interim assessment results to 
tailor instruction to individual students’ needs? 

0 = Did not use in this way; 

1 =Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Instructional Responses 

Change Scope and 
Sequence 

How much have you used the latest interim assessment results to 
identify and correct gaps in the curriculum for all students? 

0 = Did not use in this way; 

1 =Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Instructional Responses 

Provide Supplemental 
Resources to Targeted 
Students 

How much have you used the latest interim assessment results to 
recommend tutoring or other educational services for students? 

0 = Did not use in this way; 

1 =Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Instructional Responses 

Establish/Adjust 

Groupings 

How much have you used the latest interim assessment results to 
assign or reassign students to classes or groups? 

0 = Did not use in this way; 

1 =Used minimally; 

2 = Used moderately; 

3 = Used extensively 

Barriers to Data Use 

Barriers to Data Use 

To what extent has lack of time to study and think about available 
data hindered your ability to use data to make instructional 
decisions? 

0 = Not at all; 

1 =To a minor extent; 

2 =To a major extent; 

3 =To a great extent 
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Appendix C : Information on Samples 


This appendix includes information on the number of schools, principals, teachers, and students involved in the study, from 
those invited to participate to the final analytic sample. 

Exhibit C.l. Sample Tracking Flowchart 
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Teacher Analytic Samples After Random Selection 
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After Random Selection of Student-T eacher L inks 

Principals: IV =212 d 
Teachers: N =l,581 e 
Students N =61,798 f 


ir 


ir 


ir 


E lementary G rades 
Reading 

Schools: n =110 
Prinapals n =101 
Teache's: n =614 
Students: n =14,764 


E lementary G rades 
M athematics 

Schools: n =111 
Prinapals: n =102 
Teache's: n =593 
Students: n =14,354 


M iddle G rades Reading 

Schools n =85 
Principals n =75 
Teachers « =532 
Students: n =36,169 


E lementary G rades 
Reading 9 

Context Key Dimension: 

—Schools n =107 
—Teachers: n =518 
— Students n =10,761 

Supports for Data Use 
Key Dimension: 

—Schools n =107 
—Teachers: n =510 
— Students n =10,598 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Schools n =107 
—Teachers: n =523 
— Students n =10,892 

Barriers to Data Use: 

—Schools n =107 
—Teachers: n =508 
— Students n =10,557 


1 

E lementary G rades 
M athematics? 

Context Key Dimension: 

—Schools: n =108 
— Teachers n =489 
— Students: n =10,429 

Supports for Data Use 
Key Dimension: 

—Schools: n =108 
— Teachers n =488 
— Students n =10,421 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Schools: n =108 
— Teachers n =499 
— Students: n =10,642 

Barriers to Data Use: 

—Schools: n =108 
— Teachers « =486 
— Students: n =10,379 


t 

M iddle G rades Reading 9 


Context Key Dimension: 

—Schools n =85 
— Teacher^ =448 
— Students n =27,580 

Supports for Data Use 
Key Dimension: 

—Schools n =85 
—Teachers: n =433 
— Students n =26,634 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Schools n =85 
—Teachers: n =450 
—Students n =27,697 

Barriers to Data Use: 

— School n =85 
— Teachers: n =436 
— Students n =26,710 


ir 


ir 


M iddle G rades 
M athematics 

Schools n =86 
Principals: n =76 
Teachers: n =471 
Students: n =38,583 


* 

Middle Grades 
M athematics 9 

Context Key Dimension: 

—Schools n =84 
— Teachers « =412 
— Students: n =30,378 

Supports for Data Use 
Key Dimension: 

—Schools n =84 
— Teachers « =391 
— Students: n =28,902 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Schools n =84 
— Teachers « =412 
—Students: n =30,368 

Barriers to Data Use: 

— Schools n =84 
— Teachers « =387 
— Students: n =28,402 
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Principal Analytic Samples 


E lementary G rades 
Reading 9 

C ontext K ey D i mension: 

— Principals: n- 99 
— Students: n = 10,175 

Supports for Data Use 
Key Dimension: 

—Principals: n =92 
— Students: n = 9,365 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Principals: n- 96 
—Students: n =9,788 

Barriers to Data Use: 

— Principals: n- 82 
— Students: n - 8,430 


E lementary G rades 
M athematics 9 

Context Key Dimension: 

—Principals: n = 100 
— Students: n - 10,073 

Supports for Data Use 
Key Dimension: 

—Principals: n = 93 
— Students: n - 9,266 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Principals: n =97 
—Students: n = 9,690 

Barriers to Data Use: 

—Principals: n =83 
— Students: n =8,373 


Middle Grades 
Reading 9 

Context Key Dimension: 

— Principals: n = 74 
— Students: n - 23,632 

Supports for Data Use 
Key Dimension: 

—Principals: n =68 
— Students: n =21,686 

Working With Data/ 
Instructional Responses 
Key Dimension: 

— Principals: n = 74 
—Students: n =23,809 

Barriers to Data Use: 

— Principals: n = 55 
— Students: n = 18,348 


Middle Grades 
M athematics 9 

Context Key Dimension: 

— Principals: n = 74 
— Students: n = 26,708 

Supports for Data Use 
Key Dimension: 

—Principals: n =67 
— Students: n = 23,241 

Working With Data / 
Instructional Responses 
Key Dimension: 

— Principals: n = 73 
—Students: n = 26,085 

Barriers to Data Use: 

— Principals: n = 54 
— Students: n = 20,931 


a Schools across the four districts were oversampled, and 193 schools were invited to participate in the data use 
surveys. Based on power analyses, only 180 schools were needed to detect an effect size of 0.18. b Out of the 
212 administrators, 174 were principals, and 38 were assistant principals. c Across the four districts, 72 teachers who 
completed data-use surveys could not be linked with students. d Principal data were aggregated to the school level 
because some schools had multiple principals respond to the data-use survey. As a result, principal data for 
174 schools are linked with teacher data use and student achievement. e One teacher who taught 23 students was 
dropped during the random selection process. f One district had 118 students who were each linked with two 
teachers who taught both reading and mathematics. H owever, the teacher responsible for each student’s reading and 
mathematics instruction could not be determined because roster course information was not provided for either 
teacher. These students were dropped from the student sample. 9 Three teachers and 94 students were excluded from 
the analytic samples because these teachers changed schools or grade levels within the same academic year. The 
data for these teachers and students were retained for descriptive statistics only. 
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Appendix D: Estimation Methods and HypothesisTesting 


This appendix presents intra-class correlations (ICCs), a discussion of potential multi-col linearity, the results for the struc- 
tural equation models (SEM s), and the results for the hierarchical linear models (HLM s). 

• Table D. 1 shows the variance in student achievement at the student, teacher, and school levels. Tables D.2 
and D.3 show the variance in teacher data use at the teacher and school levels. 

• Exhibits D.1-D.8 present the full model results for the SEM models of general data use and student 
achievement. 

• Table D.4 presents the covariates examined in developing the HLM model and includes a list of all 
control variables included in the final model for HLM analyses. 


• Tables D.5-D.36 present the full model results for the HLM models for each data use dimension and 
student achievement. 

I ntraclass C orrelations 

Tables D.1-D.3 present ICCs based on unconditional HLM models. Table D.l shows the ICC based on an unconditional 
three-level model for student achievement (with students nested in teachers 15 nested in schools), which were estimated 
separately for elementary and middle grades and for reading and mathematics. These ICCS were examined to understand 
the percentage of variance in student achievement at each level (the student, teacher, and school level). The majority of 
variance in student achievement was at the student level (between 54 and 81 percent), with more teacher-level variance in 
middle grades (33 to 37 percent in middle grades versus 9 and 10 percent in elementary grades). Finally, school-level vari- 
ance ranged from 6 to 10 percent. 

Table D.l. Partitioning of Variance in Student Achievement at the Student, Teacher, 

and School Level 


Dependent Variable: Student-Level Academic Achievement 


E lementary 
Grades 
Reading 

E lementary 
Grades 
M athematics 

Middle 

Grades 

Reading 

Middle Grades 
M athematics 

Student-level variance 

81.15 

78.84 

54.08 

60.39 

Teacher-level variance 

9.68 

10.70 

37.56 

33.17 

School-level variance 

9.17 

10.46 

8.36 

6.44 


ICCs were also calculated to determine the variance in teachers’ data-use surveys. Specifically presented in Tables D.2 and 
D.3 is the percent of variance in teachers' data use at the teacher and school level. These ICCs were calculated on the basis 
of the results of unconditional two-level models (level 1 teachers, level 2 schools) with each teacher data-use scale as the 
outcome. The majority of variance in teachers' data use was at the teacher level, with less variance at the school level (rang- 
ing from 7 to 26 percent) depending on grade and subject. 


15 The second level of the HLM analyses represents teachers, not classrooms. Each teacher is linked with the students in their class(es). Students are linked with only one 
reading and one mathematics teacher. 
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Table D. 2. Partitioning of Variance (I CCs) in Reading Teachers' Data Use at the 

Teacher and School Level 


Dependent Variable: Reading Teacher Data Use 



1 nstructional 

Working 

with 



Context Support 

Responses 

Data 

Barriers 

Teacher level variance 

82.39 81.90 

76.34 

77.65 

88.23 

School level variance 

17.61 18.10 

23.66 

22.35 

11.77 


Table D.3. Partitioning of Variance (I CCs) in Mathematics Teachers' Data Use at the 

Teacher and School Level 


Dependent Variable: Mathematics Teacher Data Use 


Context 

Support 

1 nstructional 
Responses 

Working 
with Data 

Barriers 

Teacher level variance 

78.95 

80.65 

73.21 

74.07 

92.77 

School level variance 

21.05 

19.35 

26.79 

25.93 

7.23 


Addressing M ulticollinearity 

Following a review of the descriptive statistics and bivariate correlations among the data-use variables, we further specified 
the HLM and SEM analysis strategy to be used to address the potential issues with multicol I i nearity. There is currently no 
direct diagnostic test for multicollinearity in multilevel models. Therefore, we carried out multicollinearity diagnostics in 
a multiple regression framework, where data-use variables were independent variables, and classroom/school averages of 
student achievement were dependent variables. We used three collinearity diagnostics: VIF (variance inflation factor), TOL 
(tolerance), and COLLINOINT (which displays several different measures of collinearity). The large VIF values (greater 
than 4.0) suggest that there was a high degree of collinearity among these data-use variables. 

We then took several steps to address the potential multicollinearity in our subsequent analyses. First, we used all four key 
dimensions of data to create one latent variable of general data use to adjust for the correlations between each of the dimen- 
sions of data use in the SEM analyses. 16 Second, we estimated each dimension of data use separately for teachers and ad- 
ministrators in the H LM analyses, rather than using a model that simultaneously estimated the relationship between all key 
dimensions of data use and student achievement (which could be plagued by multicollinearity problems). Third, because 
the two scales Working with Data and Instructional Responses were consistently the most highly correlated and thus could 
be considered largely redundant, we combined them into a new variable: "Attention to Data in the Classroom" for teachers 
and "Attention to Data in the School" for principals. We used these combined attention to data variables as the predictors 
in the HLM analyses. 

Structural E quation M odels (SE M s) 

Exhibits D.1-D.8 present the path coefficients for all paths tested in the SEMs of interim assessment data use. Each exhibit 
is a separate analysis conducted with a different sample of principals, teachers, and students. Exhibits D.1-D.4 present 
teacher models of general data use, and Exhibits D.5-D. 8 present principal models of general data use. 


16 The B arriers variable was kept separate, given that its correlations with the other four data-use variables were significant but low or moderate (all below 0.50). 
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In structural equation modeling, rectangular boxes represent indicators or observed variables, and large circles or ovals 
represent latent variables. In the following sets of models, each has 10 observed variables (represented by square boxes) 
and one latent variable for “Teacher/Principal General Data Use” (represented by a large oval). Correlated errors of latent 
variables and covariances between observed variables are represented by double arrowed lines. Factor loadings and regres- 
sion paths are represented using lines with only one arrow. 


Exhibit D.l. Structural Equation M odel of the Relationship Between Teachers' General Data Useand 

Student Achievement in E lementary G rades Reading 



Note. ** p < .01, two tailed; * p < .05, two-tailed. 
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Exhibit D.2. Structural Equation M odel of the Relationship Between Teachers' General Data Useand 

Student Achievement in E lementary G rades M athematics 
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Exhibit D.3. Structural Equation M odel of the Relationship Between Teachers' General Data Useand 

Student Achievement in M iddle G rades Reading 



Note. ** p < .01, two tailed. 
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Exhibit D.4. Structural Equation M odel of the Relationship Between Teachers' General Data Useand 

Student Achievement in M iddle G rades M athematics 
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Exhibit D.5. Structural Equation Model of the Relationship Between Principals' General Datallseand 

Student Achievement in E lementary G rades Reading 



Note. ** p < .01, two-tailed. 
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Exhibit D.6. Structural Equation Model of the Relationship Between Principals' General Datallseand 

Student Achievement in E lementary G rades M athematics 
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Exhibit D.7. Structural Equation Model of the Relationship Between Principals' General Datallseand 

Student Achievement in M iddle G rades Reading 
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Exhibit D.8. Structural Equation Model of the Relationship Between Principals' General Datallseand 

Student Achievement in M iddle G rades M athematics 
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H ierarchical L inear M odels (H L M s) 

C entering of Variables 

The measures of data use and student achievement used in the study, like many psychological and educational constructs, 
are expressed on arbitrary metrics thatlack a meaningful zero point. In such situations, centering isoften used in estimating 
hierarchical models, to facilitate interpretation of the intercept (Enders & Tofighi, 2007). We used grand-mean centering for 
student prior achievement, student demographics, and teacher and principal demographic and teaching experience covari- 
ates (Luke, 2004). In other words, all student-level variables and teacher and principal covariates were centered to have a 
mean of zero across the full sample. Teacher data-use variables were group-mean centered (i.e., centered on the school aver- 
ages). Principal data-use variables were centered at the grand mean because their scores are constant for each school. These 
centering decisions are consistent with the guidelines proposed by Enders and Tofighi (2007). 

E xamining and Selecting C ovariates 

Before estimating the H L M models to address the research questions, we estimated a set of exploratory three- and two-level 
models that included student-, teacher-, and school-level covariates to develop a model of important covariates that should 
be included in the final model. To estimate the control model, we tested each set of covariates separately, beginning with 
level 1 covariates (student-level variables), followed by level 2 covariates (teacher-level variables), and finally adding level 
3 covariates (principal-/school-level variables). All control models also included district indicators, grade-level indicators, 
prior achievement, and their two-way or three-way interactions. The final control model contains the set of district, grade- 
level indicators, and all significant covariates from among those listed. Variables tested in the control model and those kept 
in the final control model are presented in Table D.4. 

HLM Final Model Results 

Tables D.5-D.36 present the full results for the HLM models of the relations of each data-use dimension and student 
achievement. These tables all are organized into two main sections, with the first one for fixed effects and the second one 
for random effects. The Fixed Effect section summarizes unstandardized regression coefficients and standard errors cor- 
responding to the data-use variable and control variables (e.g., district-, school-, teacher-, and student-level covariates, and 
the interaction terms). 

Tables D.5-D.12 present the results for the HLM models of teacher data use and student achievement in mathematics and 
reading at the elementary grade level. Tables D.13-D.20 present the H L M results of teacher data use and student achieve- 
ment in mathematics and reading in middle school grades. Tables D.21-D.36 present results of principal data use and stu- 
dent achievement in the two subject matters at both middle and elementary schools. 
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Table D.4. Covariates Tested in Developing and Included in the F inal Control Model 



Variables Included in the 
Variables Tested final Control Model 

C ovariates 

District indicators 

V 

V 

Grade level 

V 

V 

Prior achievement (reading and mathematics) 

V 

V 

District by grade interactions 

V 

V 

District by prior achievement interactions 

V 

V 

Student Demographics 

Gender 

V 

V 

Hispanic 

V 

V 

Asian (including Pacific Islander) 

V 

V 

African American 

V 

V 

Native American 

V 

V 

M ulti racial 

V 

V 

Prior reading achievement 

V 

V 

F ree or reduced-price lunch status 

V 

V 

Special education status 

V 

V 


Variables 

Variables 1 nduded in the 


Tested 

Final Control Model 

Teacher Covariates 

Gender 

V 

V 

Hispanic 

V 


Asian (including Pacific Islander) 

V 


African American 

V 


Native American 

V 


Other racial 

V 


Level of education 

V 

V 

Y ears of teaching 

V 

V 

C lassroom average student achievement scores 

V 

v a 

Principal Covariates 

Gender 

V 


Hispanic 

V 


Asian (including Pacific Islander) 

V 


African American 

V 


Native American 

V 


Other racial groups 

V 


Level of education 

V 


Y ears of teaching 

V 


Y ears of administration 

V 


School average student achievement scores 

V 

v b 


a Classroom average of student achievement was included in the HLM models only when teacher data use was the 
predictor. 

“School average of student achievement was included in the models only when principal reported data use was the 
predictor. 
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APPENDICES CONT’D 


Table D.5. Estimated Relationship Between ContBctforDalaUseAmong Elementary 
School Teachers and Student Achievement in M athematics (N=10,429) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.022 

.048 

.652 

Contact for Data Use 3 

.034 

.044 

.435 

District covariates 




District 1 ( versus District 3) 

-.017 

.066 

.792 

District 2 ( versus District 3) 

-.035 

.063 

.581 

District 4 ( versus District 3) 

.090 

.068 

.186 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.299 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.068 

.029 

.018 

Female 

.022 

.036 

.545 

Classroom- level prior mathematics achievement 13 

.063 

.032 

.052 

Student-level covariates 




Prior mathematics achievement 13 

.714 

.020 

< .0001 

Grade 5 (versus Grade 4) 

.008 

.047 

.861 

Hispanic (versus white) 

-.055 

.018 

.002 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.092 

.030 

.002 

African American ( versus white) 

-.139 

.020 

< .0001 

American Indian (versus white) 

-.111 

.051 

.029 

M ulti racial (versus white) 

CO 

o 
1— 1 

.093 

.243 

Eligible for free or reduced-price lunch 

-.120 

.016 

< .0001 

Female 

.014 

.012 

.226 

IEP 

-.035 

.018 

.061 

Interaction terms 




District 1 x Grade 5 

-.032 

.065 

.622 

District 2 x Grade 5 

-.046 

.064 

.473 

District 4 x Grade 5 

-.033 

.077 

.671 

District 1 x prior mathematics achievement 

-.086 

.027 

.001 

District 2 x prior mathematics achievement 

-.023 

.025 

.353 

District 4 x prior mathematics achievement 

-.166 

.031 

< .0001 

Grade 5 x prior mathematics achievement 

.200 

.028 

< .0001 

District 1 x Grade 5 x prior mathematics achievement 

-.212 

.038 

< .0001 

District 2 x Grade 5 x prior mathematics achievement 

-.129 

.035 

0 

District 4 x Grade 5 x prior mathematics achievement 

-.151 

.045 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.364 

.005 

< .0001 

Level 2 

.053 

.005 

< .0001 

Level 3 

.020 

.006 

0 

Total variance 

.438 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay, Perez -Johnson, Haimson, Sattar, & Gleason, 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.6. Estimated Relationship Between Supportsfor Data UseAmong Elementary 
School Teachers and Student Achievement in M athematics (N=10,421) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.018 

.046 

.693 

Supportsfor Data Use 3 

.008 

.031 

.790 

District covariates 




District 1 (versus District 3) 

-.011 

.064 

.863 

District 2 (versus District 3) 

-.034 

.062 

.582 

D i stri ct 4 ( versus D i strict 3) 

.103 

.066 

.123 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.246 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.076 

.029 

.008 

Female 

.024 

.036 

.494 

Classroom-level prior mathematics achievement 13 

.068 

.032 

.032 

Student-level covariates 




Prior mathematics achievement 0 

.718 

.020 

< .0001 

G rade 5 ( versus G rade 4) 

.017 

.045 

.701 

Hispanic (versus white) 

-.058 

.018 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.089 

.030 

.003 

African American (versus white) 

-.143 

.019 

< .0001 

A merican 1 ndian (versus white) 

-.120 

.051 

.018 

M ulti racial (versus white) 

-.125 

.087 

.151 

Eligible for free or reduced-price lunch 

-.118 

.017 

< .0001 

Female 

.011 

.012 

.346 

IEP 

-.039 

.018 

.036 

Interaction terms 




District 1 x Grade 5 

i — 1 

LO 

O 

i 

.063 

.414 

District 2 x Grade 5 

-.045 

.063 

.471 

District 4 x Grade 5 

-.049 

.075 

.516 

District 1 x prior mathematics achievement 

-.089 

.026 

.001 

District 2 x prior mathematics achievement 

-.023 

.024 

.353 

District 4 x prior mathematics achievement 

-.160 

.031 

< .0001 

Grade 5 x prior mathematics achievement 

.194 

.027 

< .0001 

District 1 x Grade 5 x prior mathematics achievement 

-.208 

.037 

< .0001 

District 2 x Grade 5 x prior mathematics achievement 

-.128 

.034 

0 

District 4 x Grade 5 x prior mathematics achievement 

-.155 

.044 

0 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.360 

.005 

< .0001 

Level 2 

.051 

.005 

< .0001 

Level 3 

0020 

.006 

0 

Total variance 

.431 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D.7. Estimated Relationship Between Attention to Data in the Classroom Among 
E lementary School Teachers and Student Achievement in M athematics (N =10,642) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.010 

.047 

.831 

Working with Data/Instructional Response 3 

.036 

.030 

.235 

District covariates 




D i stri ct 1 ( versus D i stri ct 3 ) 

-.004 

.064 

.955 

D i stri ct 2 ( versus D i stri ct 3 ) 

-.014 

.062 

.825 

D i stri ct 4 ( versus D i stri ct 3 ) 

.093 

.066 

.166 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.335 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.069 

.029 

.015 

Female 

.012 

.035 

.726 

Classroom-level prior mathematics achievement 13 

.064 

.031 

.038 

Student-level covariates 




Prior mathematics achievement 13 

.718 

.019 

<.0001 

Grade 5 [versus Grade 4) 

.018 

.045 

.688 

Hispanic [versus white) 

-.056 

.018 

0002 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.091 

.029 

.002 

African American [versus white) 

-.144 

.019 

<.0001 

A merican 1 ndian [versus white) 

-.117 

.050 

.020 

M ulti racial [versus white) 

-.126 

.088 

.152 

E ligible for free or reduced-price lunch 

-.119 

.016 

<.0001 

Female 

.013 

.012 

.266 

IEP 

-.030 

.018 

.096 

Interaction terms 




District 1 x Grade 5 

-.041 

.063 

.515 

District 2 x Grade 5 

-.061 

.062 

.330 

District 4 x Grade 5 

-.044 

.075 

.564 

District 1 x prior mathematics achievement 

-.090 

.026 

.001 

District 2 x prior mathematics achievement 

-.027 

.024 

.264 

District 4 x prior mathematics Achievement 

-.170 

.030 

<.0001 

Grade 5 x prior mathematics achievement 

.195 

.027 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

-.207 

.037 

<.0001 

District 2 x Grade 5 x prior mathematics achievement 

-.125 

.034 

0 

District 4 x Grade 5 x prior mathematics achievement 

0.147 

.044 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.361 

.005 

<.0001 

Level 2 

.053 

.005 

<.0001 

Level 3 

.020 

.006 

0 

Total variance 

.434 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom- level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.8. Estimated Relationship Between BanierstoDataUseAmong 
E lementary School Teachers and Student Achievement in M athematics (N=10,379) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.017 

.046 

.718 

Barriers to Data Use a 

-.079 

.032 

.014 

District covariates 




District 1 District 3) 

-.007 

.064 

.914 

District 2 District 3) 

-.037 

.062 

.552 

District 4 ( versus District 3) 

.113 

.067 

.094 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.213 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.080 

.028 

.004 

Female 

.018 

.035 

.604 

Classroom-level prior mathematics achievement 6 

.077 

.031 

.014 

Student-level covariates 




Prior mathematics achievement 6 

.716 

.020 

<.0001 

Grade 5 (versus G rade 4) 

.016 

.044 

.721 

Hispanic (versus white) 

-.062 

.018 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.086 

.030 

.004 

African American (versus white) 

-.145 

.019 

<.0001 

A merican 1 ndian (versus white) 

-.123 

.051 

.015 

M ulti racial (versus white) 

-.132 

.088 

.135 

E ligible for free or reduced-price lunch 

-.116 

.017 

<.0001 

Female 

.012 

.012 

.299 

IEP 

-.040 

.018 

.028 

Interaction terms 




District 1 x Grade 5 

-.052 

.062 

.398 

District 2 x Grade 5 

.040 

.061 

.513 

District 4 x Grade 5 

-.091 

.074 

.219 

District 1 x prior mathematics achievement 

-.088 

.026 

.001 

District 2 x prior mathematics achievement 

-.022 

.024 

.366 

District 4 x prior mathematics achievement 

-.159 

.031 

<.0001 

Grade 5 x prior mathematics achievement 

.195 

.027 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

-.210 

.037 

<.0001 

District 2 x Grade 5 x prior mathematics achievement 

-.131 

.034 

0 

District 4 x Grade 5 x prior mathematics achievement 

-.144 

.044 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.360 

.005 

<.0001 

Level 2 

.048 

.005 

<.0001 

Level 3 

.021 

.006 

<.0001 

T otal variance 

.429 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 
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APPENDICES CONT’D 


Table D.9. Estimated Relationship Between ContBctforDataUseAmong 
Elementary School Teachers and Student Achievement in Reading (N=10,761) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.019 

.039 

.640 

Context for Data Use a 

.031 

.037 

.398 

District covariates 




District 1 (versus District 3) 

-.009 

.055 

.876 

District 2 (versus District 3) 

-.052 

.052 

.324 

D i stri ct 4 ( versus District 3) 

.045 

.056 

.424 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.001 

.115 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.025 

.025 

.328 

Female 

-.015 

.031 

.633 

Classroom-level prior reading achievement 13 

.158 

.030 

<.0001 

Student-level covariates 




Prior reading achievement 13 

.652 

.020 

<.0001 

Grade 5 (versus Grade 4) 

-.005 

.044 

.910 

H ispanic (versus white) 

-.061 

.018 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.008 

.031 

.783 

African American (versus white) 

-.137 

.020 

<.0001 

American Indian (versus white) 

-.284 

.052 

<.0001 

M ulti racial (versus white) 

-.101 

.091 

.267 

Eligible for free or reduced-price lunch 

-.107 

.017 

<.0001 

Female 

.132 

.012 

<.0001 

IEP 

-.254 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.017 

.060 

.781 

District 2 x Grade 5 

.002 

.059 

.974 

District 4 x Grade 5 

-.006 

.067 

.933 

District 1 x prior reading achievement 

-.032 

.027 

.240 

District 2 x prior reading achievement 

.039 

.025 

.113 

District 4 x prior reading achievement 

-.108 

.030 

0 

Grade 5 x prior reading achievement 

.247 

.028 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.320 

.038 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.203 

.035 

<.0001 

D istrict 4 x Grade 5 x prior reading achievement 

-.275 

.044 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.386 

.005 

<.0001 

Level 2 

.040 

.005 

<.0001 

Level 3 

.010 

.004 

.004 

Total variance 

.437 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.10. Estimated Relationship Between SupportsforDataUseAmong 
E lementary School Teachers and Student Achievement in Reading (N=10,598) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.016 

.039 

.681 

Supports for Data Use a 

-.008 

.028 

.764 

District covariates 




District 1 District 3) 

-.007 

.054 

.896 

District 2 District 3) 

-.049 

.052 

.348 

D i stri ct 4 ( versus D i stri ct 3 ) 

.065 

.056 

.248 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.001 

.083 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.033 

.025 

.198 

Female 

-.014 

.031 

.656 

Classroom-level prior reading achievement 13 

.163 

.030 

<.0001 

Student-level covariates 




Prior reading achievement 0 

.654 

.020 

<.0001 

Grade 5 [versus Grade 4) 

-.004 

.043 

.927 

Hispanic [versus white) 

-.058 

.019 

.002 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.013 

.031 

.683 

African American [versus white) 

-.135 

.020 

<.0001 

American Indian [versus white) 

-.283 

.052 

<.0001 

M ulti racial [versus white) 

-.107 

.091 

.243 

E ligible for free or reduced-price lunch 

-.106 

.017 

<.0001 

Female 

.131 

.012 

<.0001 

IEP 

-.255 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.019 

.059 

.747 

District 2 x Grade 5 

.004 

.059 

.944 

District 4 x Grade 5 

-.024 

.067 

.715 

District 1 x prior reading achievement 

-.035 

.027 

.198 

District 2 x prior reading achievement 

.048 

.025 

.056 

District 4 x prior reading achievement 

-.115 

.031 

0 

Grade 5 x prior reading achievement 

.250 

.027 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.323 

.038 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.216 

.035 

<.0001 

D i stri ct 4 x Grade 5 x prior reading achievement 

-.273 

.044 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.386 

.005 

<.0001 

Level 2 

.039 

.005 

<.0001 

Level 3 

.010 

.004 

.004 

Total variance 

.435 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D. 11. Estimated Relationship Between Attention to Data in the Classroom Among 
Elementary School Teachers and Student Achievement in Reading (N =10,892) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.005 

.039 

.901 

Working with Data/Instructional Response 3 

.063 

.026 

.014 

District covariates 




District 1 (versus District 3) 

.008 

.054 

.885 

District 2 District 3) 

-.028 

.052 

.600 

D i stri ct 4 ( versus D i stri ct 3 ) 

.039 

.056 

.486 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.001 

.165 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.026 

.025 

.300 

Female 

-.021 

.031 

.494 

Classroom-level prior reading achievement 13 

.164 

.030 

< .0001 

Student-level covariates 




Prior reading achievement 13 

.653 

.020 

< .0001 

Grade 5 (versus Grade 4) 

-.008 

.043 

.850 

H ispanic (versus white) 

-.062 

.018 

.001 

Asian/Native Hawaiian /Pacifi C Islander ( versus white) 

-.010 

.030 

.751 

African American (versus white) 

-.137 

.019 

<.0001 

A merican 1 ndian (versus white) 

-.284 

.051 

<.0001 

M ulti racial (versus white) 

-.104 

.090 

.249 

Eligible for free or reduced-price lunch 

-.106 

.016 

<.0001 

Female 

.134 

.012 

<.0001 

IEP 

-.247 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.010 

.059 

.868 

District 2 x Grade 5 

.002 

.058 

.972 

District 4 x Grade 5 

.001 

.067 

.988 

District 1 x prior reading achievement 

-.032 

.027 

.227 

District 2 x prior reading achievement 

.035 

.024 

.154 

District 4 x prior reading achievement 

-.108 

.030 

0 

Grade 5 x prior reading achievement 

.252 

.027 

< .0001 

District 1 x Grade 5 x prior reading achievement 

-.326 

.037 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.204 

.034 

<.0001 

District 4 x Grade 5 x prior reading achievement 

-.280 

.043 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.385 

.005 

<.0001 

Level 2 

.039 

.004 

<.0001 

Level 3 

.010 

.004 

.003 

T otal variance 

.434 




The key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al ., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D. 12. Estimated Relationship Between BamerstoDaftallseAmong 
Elementary School Teachers and Student Achievement in Reading (N=10 r 557) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.016 

.039 

.681 

Barriers to Data Use 3 

-.019 

.029 

.524 

District covariates 




District 1 (versus District 3) 

-.006 

.054 

.906 

District 2 (versus District 3) 

-.049 

.052 

.346 

D i stri ct 4 ( versus D i strict 3) 

.068 

.056 

.223 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.001 

.082 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.033 

.025 

.192 

Female 

-.015 

.031 

.631 

Classroom-level prior reading achievement 13 

.162 

.030 

<.0001 

Student-level covariates 




Prior reading achievement 13 

.654 

.020 

<.0001 

G rade 5 (versus G rade 4) 

-.003 

.043 

.942 

H ispanic (versus white) 

-.060 

.019 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.015 

.031 

.636 

African American (versus white) 

-.136 

.020 

<.0001 

A merican 1 ndian (versus white) 

-.284 

.052 

<.0001 

M ulti racial (versus white) 

-.112 

.092 

.223 

Eligible for free or reduced-price lunch 

I — 1 

O 

or 

.017 

<.0001 

Female 

.131 

.012 

<.0001 

IEP 

-.255 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.019 

.059 

.752 

District 2 x Grade 5 

.001 

.059 

.983 

District 4 x Grade 5 

-.042 

.067 

.535 

District 1 x prior reading achievement 

-.034 

.027 

0204 

District 2 x prior reading achievement 

.046 

.025 

.063 

District 4 x prior reading achievement 

-.115 

.031 

0 

Grade 5 x prior reading achievement 

.252 

.027 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.325 

.038 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.218 

.035 

<.0001 

District 4 x Grade 5 x prior reading achievement 

-.266 

.044 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.387 

.005 

<.0001 

Level 2 

.039 

.005 

<.0001 

Level 3 

.009 

.004 

.005 

Total variance 

.435 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D. 13. Estimated Relationship Between ContBctforDataUseAmong 
M iddle School Teachers and Student Achievement in M athematics (N=30 r 378) 


Fixed Effect 

Coefficient 

Std Error 

p Value 

1 ntercept 

.013 

.039 

.745 

C ontext for Data U se a 

.035 

.040 

.382 

District covariates 




District 1 (versus District 3) 

.007 

.048 

.893 

D i stri ct 2 ( versus D i stri ct 3 ) 

.002 

.046 

.958 

District 4 (versus District 3) 

.290 

.072 

.000 

Teacher-level covariates 




T otal years of teaching experience 

.000 

.002 

.749 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.024 

.029 

.405 

Female 

.015 

.029 

.599 

Classroom-level prior mathematics achievement 13 

.298 

.021 

<.0001 

Student-level covariates 




Prior mathematics achievement 13 

.750 

.012 

<.0001 

Grade 8 (versus G rade 7) 

.065 

.045 

.149 

Hispanic (versus white) 

-.103 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.079 

.015 

<.0001 

African American (versus white) 

-.125 

.011 

<.0001 

American Indian (versus white) 

-.096 

.033 

.003 

M ulti racial (versus white) 

-.047 

.079 

.552 

Eligible for free or reduced-price lunch 

-.094 

.009 

<.0001 

Female 

-.036 

.007 

<.0001 

IEP 

.002 

.013 

.881 

Interaction terms 




District 1 x Grade 8 

-.230 

.052 

<.0001 

District 2 x Grade 8 

CO 

o 
1 — 1 

.049 

.027 

District 4 x Grade 8 

-.086 

.076 

.258 

District 1 x prior mathematics achievement 

-.305 

.016 

<.0001 

District 2 x prior mathematics achievement 

O 
1— » 

.014 

.237 

District 4 x prior mathematics achievement 

-.195 

.031 

<.0001 

Grade 8 x prior mathematics achievement 

.054 

.017 

.002 

District 1 x Grade 8 x prior mathematics achievement 

-.093 

.023 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.113 

.020 

<.0001 

District 4 x Grade 8 x prior mathematics achievement 

-.108 

.049 

.028 

Random Effect 

Variance 

Std Error 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.055 

.005 

<.0001 

Level 3 

.002 

.002 

.204 

Total variance 

.404 




a The key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom- level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


TableD.14. Estimated Relationship Between Supports for Data UseAmong 
M iddle School Teachers and Student Achievement in M athematics (N=28,902) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.019 

.040 

.629 

Supports for Data Use 3 

-.005 

.035 

.892 

District covariates 




D i Stri Ct 1 ( versus D i Stri Ct 3 ) 

0 

.049 

.998 

D i Stri Ct 2 ( versus D i Stri Ct 3 ) 

-.003 

.047 

.941 

D i Stri Ct 4 ( versus D i Stri Ct 3 ) 

.286 

.075 

0 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.002 

.672 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.024 

.030 

.419 

Female 

.011 

.030 

.704 

Classroom- lev el prior mathematics achievement 11 

.298 

.021 

<.0001 

Student-level covariates 




Prior mathematics achievement 0 

.752 

.012 

<.0001 

G rade 8 (versus G rade 7) 

.058 

.045 

.206 

Hispanic (versus white) 

-.106 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.074 

.016 

<.0001 

African American (versus white) 

-.126 

.012 

<.0001 

American Indian (versus white) 

-.093 

.034 

.006 

M ulti racial (versus white) 

-.052 

.083 

.532 

Eligible for free or reduced-price lunch 

-.094 

.009 

<.0001 

Female 

-.035 

.007 

<.0001 

IEP 

.008 

.013 

.557 

Interaction terms 




District 1 x Grade 8 

-.241 

.053 

<.0001 

District 2 x Grade 8 

-.097 

.049 

.048 

D i stri ct 4 xGrade8 

-.097 

.078 

.212 

District 1 x prior mathematics achievement 

-.313 

.017 

<.0001 

District 2 x prior mathematics achievement 

-.019 

.014 

.192 

District 4 x prior mathematics achievement 

-.212 

.032 

<.0001 

G rade 8 x prior mathematics achievement 

.043 

.018 

.014 

District 1 x Grade 8 x prior mathematics achievement 

-.101 

.024 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.105 

.021 

<.0001 

District 4 x Grade 8 x prior mathematics achievement 

-.033 

.053 

.532 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.346 

.003 

<.0001 

Level 2 

.056 

.005 

<.0001 

Level 3 

.002 

.003 

.210 

Total variance 

.404 




^he key dimensions of data use for teachers were group-mean centered. b V alues are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 
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APPENDICES CONT’D 


Table D. 15. Estimated Relationship Between Attention to Data in the Classroom Among 
M iddle School Teachers and Student Achievement in M athematics (N =30,368) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.011 

.039 

.776 

Working with Data/Instructional Response 3 

.093 

.026 

.000 

District covariates 




District 1 (versus District 3) 

.012 

.047 

.793 

District 2 (versus District 3) 

.024 

.045 

.593 

D i stri ct 4 ( versus D i stri ct 3 ) 

.257 

.072 

.001 

Teacher-level covariates 




T otal years of teaching experience 

.000 

.002 

.794 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.020 

.028 

.491 

Female 

.025 

.029 

.377 

Classroom- level prior mathematics achievement 11 

.306 

.020 

<.0001 

Student-level covariates 




Prior mathematics achievement 

.750 

.012 

<.0001 

Grade 8 (versus Grade 7) 

.065 

.045 

.147 

Hispanic (versus white) 

-.103 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.079 

.015 

<.0001 

African American (versus white) 

-.125 

.011 

<.0001 

American Indian (versus white) 

-.095 

.033 

.003 

M ulti racial (versus white) 

-.047 

.079 

.551 

Eligible for free or reduced-price lunch 

-.095 

.009 

<.0001 

Female 

-.036 

.007 

<.0001 

IEP 

.003 

.013 

.813 

Interaction terms 




District 1 x Grade 8 

-.226 

.052 

<.0001 

District 2 x Grade 8 

-.108 

.049 

.027 

District 4 x Grade 8 

-.089 

.076 

.238 

District 1 x prior mathematics achievement 

-.306 

.016 

<.0001 

District 2 x prior mathematics achievement 

-.017 

.014 

.230 

District 4 x prior mathematics achievement 

-.194 

.031 

<.0001 

Grade 8 x prior mathematics achievement 

.053 

.017 

.002 

District 1 x Grade 8 x prior mathematics achievement 

-.092 

.023 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.113 

.020 

<.0001 

District 4 x Grade 8 x prior mathematics achievement 

-.108 

.049 

.027 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.054 

.005 

<.0001 

Level 3 

.001 

.002 

.253 

Total variance 

.403 




^The key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D. 16. Estimated Relationship Between BamerstoDatallseAmong 
M iddle School Teachers and Student Achievement in M athematics (N=28,402) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.023 

.040 

.565 

Barriers to Data Use 3 

-.037 

.033 

.267 

District covariates 




District 1 ( versus District 3) 

-.006 

.050 

.906 

District 2 (versus District 3) 

-.005 

.047 

.922 

District 4 (versus District 3) 

.281 

.075 

.000 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.002 

.665 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.017 

.030 

.559 

Female 

.003 

.030 

.926 

Classroom- level prior mathematics achievement 11 

.303 

.021 

<.0001 

Student-level covariates 




Prior mathematics achievement 

.752 

.012 

<.0001 

Grade 8 (versus Grade 7) 

.055 

.045 

.222 

Hispanic (versus white) 

-.104 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.074 

.016 

<.0001 

African American (versus white) 

-.126 

.012 

<.0001 

American Indian (versus white) 

-.102 

.034 

.003 

M ultiracial (versus white) 

-.051 

.082 

.532 

Eligible for free or reduced-price lunch 

-.094 

.009 

<.0001 

Female 

-.037 

.007 

<.0001 

IEP 

.011 

.013 

.397 

Interaction terms 




District 1 x Grade 8 

-.235 

.053 

<.0001 

District 2 x Grade 8 

-.101 

.049 

.041 

District 4 x Grade 8 

-.094 

.077 

.223 

District 1 x prior mathematics achievement 

-.303 

.017 

<.0001 

District 2 x prior mathematics achievement 

-.018 

.014 

.217 

District 4 x prior mathematics achievement 

-.213 

.031 

<.0001 

Grade 8 x prior mathematics achievement 

.043 

.018 

.016 

District 1 x Grade 8 x prior mathematics achievement 

-.110 

.024 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.103 

.021 

<.0001 

District 4 x Grade 8 x prior mathematics achievement 

-.031 

.053 

.555 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.345 

.003 

<.0001 

Level 2 

.054 

.005 

<.0001 

Level 3 

.003 

.003 

.152 

Total variance 

.401 




^he key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate 
the scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 
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Table D. 17. Estimated Relationship Between ContBctforDatallseAmong 
M iddle School Teachers and Student Achievement in Reading (N=27 r 580) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.083 

.040 

.041 

C ontext for Data U se a 

.057 

.031 

.067 

District covariates 




District 1 ( versus District 3) 

-.081 

.052 

.121 

D i stri ct 2 ( versus D i stri ct 3) 

-.076 

.048 

.121 

D i Stri Ct 4 ( versus D i Stri Ct 3) 

.010 

.073 

.888 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.001 

.608 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.023 

.027 

.379 

Female 

.048 

.032 

.135 

Classroom- level prior reading achievement 13 

.285 

.018 

<.0001 

Student-level covariates 




Prior reading achievement 

.695 

.012 

<.0001 

Grade 8 ( versus Grade 7) 

-.020 

.041 

.630 

H ispanic ( versus white) 

-.076 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.002 

.015 

.921 

African American (versus white) 

-.125 

.012 

<.0001 

American Indian (versus white) 

-.144 

.033 

<.0001 

M ultiracial (versus white) 

-.073 

.082 

.371 

Eligiblefor free or reduced-price lunch 

-.091 

.009 

<.0001 

Female 

.114 

.007 

<.0001 

IEP 

-.224 

.014 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.071 

.049 

.143 

District 2 x Grade 8 

.047 

.046 

.315 

District 4 x Grade 8 

.034 

.067 

.614 

District 1 x prior reading achievement 

-.206 

.018 

<.0001 

District 2 x prior reading achievement 

.017 

.015 

.238 

District 4 x prior reading achievement 

-.093 

.029 

.001 

Grade 8 x prior reading achievement 

-.006 

.019 

.758 

District 1 x Grade 8 x prior reading achievement 

-.043 

.025 

.087 

District 2 x Grade 8 x prior reading achievement 

.011 

.022 

.630 

District 4 x Grade 8 x Prior reading achievement 

-.045 

.040 

.259 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.342 

.003 

<.0001 

Level 2 

.039 

.004 

<.0001 

Level 3 

.010 

.003 

.001 

Total variance 

.392 




T'he key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D. 18. Estimated Relationship Between SinxxIsforDartaUseAmong 
M iddle School Teachers and Student Achievement in Reading (N=26,634) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.086 

.040 

.032 

Supports for Data Use a 

.010 

.030 

.751 

District covariates 




District 1 ( versus District 3) 

-.082 

.051 

.115 

D i Stri Ct 2 ( versus D i Stri Ct 3) 

-.088 

.048 

.070 

D i stri ct 4 ( versus D i stri ct 3) 

.005 

.073 

.944 

Teacher-level covariates 




T otal years of teaching experience 

0 

.001 

.806 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.024 

.027 

.370 

Female 

.062 

.033 

.059 

Classroom- level prior reading achievement 13 

.285 

.018 

<.0001 

Student-level covariates 




Prior reading achievement 0 

.695 

.012 

<.0001 

Grade 8 ( versus Grade 7) 

-.024 

.041 

.553 

Hispanic (versus white) 

-.078 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.003 

.016 

.828 

African American (versus white) 

-.128 

.012 

<.0001 

American Indian (versus white) 

-.153 

.034 

<.0001 

M ulti racial (versus white) 

-.074 

.082 

.364 

Eligible for free or reduced-price lunch 

-.087 

.010 

<.0001 

Female 

.115 

.007 

<.0001 

IEP 

-.221 

.014 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.070 

.049 

.153 

District 2 x Grade 8 

.067 

.047 

.151 

District 4 x Grade 8 

.027 

.068 

.697 

District 1 x prior reading achievement 

-.195 

.018 

<.0001 

District 2 x prior reading achievement 

.015 

.015 

.299 

District 4 x prior reading achievement 

-.088 

.030 

.003 

Grade 8 x prior reading achievement 

-.004 

.019 

.842 

District 1 x Grade 8 x prior reading achievement 

-.062 

.026 

.016 

District 2 x Grade 8 x prior reading achievement 

.016 

.022 

.473 

District 4 x Grade 8 x prior reading achievement 

-.069 

.041 

.091 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.343 

.003 

<.0001 

Level 2 

.041 

.005 

<.0001 

Level 3 

.009 

.003 

.003 

Total variance 

.393 




‘The key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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Table D. 19. Estimated Relationship Between Attention to Data in the Classroom Among 
Middle School Teachers and Student Achievement in Reading (N =27,697) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.081 

.040 

.047 

Working with Data/I nstructional Response 3 

.017 

.023 

.467 

District covariates 




D i Stri Ct 1 (versus D i Stri Ct 3) 

-.077 

.051 

.141 

D i Stri Ct 2 (versus D i Stri Ct 3) 

-.072 

.049 

.143 

D i Stri Ct 4 (versus D i Stri Ct 3) 

.005 

.072 

.941 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.001 

.586 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.024 

.027 

.364 

Female 

.052 

.032 

.104 

Classroom- level prior reading achievement 13 

.283 

.018 

<.0001 

Student-level covariates 




Prior reading achievement 0 

.695 

.012 

<.0001 

Grade 8 (versus Grade 7) 

-.021 

.041 

.610 

Hispanic (versus white) 

-.076 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander (versus white) 

.002 

.015 

.895 

African American (versus white) 

-.124 

.012 

<.0001 

American Indian (versus white) 

-.144 

.033 

<.0001 

M ulti racial (versus white) 

-.073 

.082 

.374 

E 1 i g i bl e for free or reduced- pri ce 1 unch 

-.091 

.009 

<.0001 

Female 

.115 

.007 

<.0001 

IEP 

-.225 

.014 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.072 

.048 

.137 

District 2 x Grade 8 

.048 

.046 

.295 

D i stri ct 4 x Grade 8 

.038 

.067 

.572 

District 1 x prior reading achievement 

-.205 

.018 

<.0001 

District 2 x prior reading achievement 

.017 

.015 

.241 

District 4 x prior reading achievement 

-.094 

.029 

.001 

Grade 8 x prior reading achievement 

-.004 

.019 

.828 

District 1 x Grade 8 x prior reading achievement 

-.046 

.025 

.066 

District 2 x Grade 8 x prior reading achievement 

.009 

.022 

.679 

District 4 x Grade 8 x prior reading achievement 

-.047 

.040 

.242 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.342 

.003 

<.0001 

Level 2 

.040 

.004 

<.0001 

Level 3 

.010 

.003 

.001 

T otal variance 

.392 




^he key dimensions of data use for teachers were group-mean centered. b Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.20. Estimated Relationship Between Barriersto Data UseA mong 
M iddle School Teachers and Student Achievement in Reading (N=26,710) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.087 

.040 

.034 

Barriersto Data Use 3 

-.002 

.028 

.957 

District covariates 




District 1 (versus District 3) 

-.086 

.052 

.102 

District 2 (versus District 3) 

-.088 

.049 

.074 

District 4 (versus District 3) 

.015 

.074 

.838 

Teacher-level covariates 




T otal years of teaching experience 

0 

.001 

.890 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.027 

.027 

.331 

Female 

.063 

.033 

.059 

Classroom- level prior reading achievement 13 

.284 

.018 

<.0001 

Student-level covariates 




Prior reading achievement 13 

.695 

.012 

<.0001 

Grade 8 (verms Grade 7) 

-.024 

.041 

.557 

Hispanic (versus white) 

-.077 

.010 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.005 

.016 

.749 

African American (versus white) 

-.128 

.012 

<.0001 

American Indian (versus white) 

-.146 

.034 

<.0001 

M ultiracial (versus white) 

-.074 

.082 

.365 

Eligible for free or reduced-price lunch 

-.089 

.010 

<.0001 

Female 

.116 

.007 

<.0001 

IEP 

-.224 

.014 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.071 

.049 

.153 

District 2 x Grade 8 

.068 

.047 

.149 

District 4 x Grade 8 

.026 

.069 

.709 

District 1 x prior reading achievement 

-.208 

.018 

<.0001 

District 2 x prior reading achievement 

.015 

.015 

.301 

District 4 x prior reading achievement 

-.094 

.030 

.002 

Grade 8 x prior reading achievement 

-.004 

.019 

.845 

District 1 x Grade 8 x prior reading achievement 

-.050 

.025 

.049 

District 2 x Grade 8 x prior reading achievement 

.015 

.022 

.481 

District 4 x Grade 8 x prior reading achievement 

-.060 

.041 

.145 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.344 

.003 

<.0001 

Level 2 

.042 

.005 

<.0001 

Level 3 

.010 

.003 

.002 

Total variance 

.395 




‘‘The key dimensions of data use for teachers were group-mean centered. “Values are standardized classroom-level aggregates of 
students’ prior achievement. Each teacher has one unique value for the average of their students’ prior achievement from the 
2009 state reading assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 
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APPENDICES CONT’D 


Table D. 21. Estimated Relationship Between ContedforDataUseAmong 
Elementary School Principals and Student Achievement in M athematics (N=10,073) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.014 

.048 

.111 

Context for Data Use a 

.086 

.060 

.152 

District covariates 




District 1 (versus District 3) 

.016 

.069 

.818 

District 2 (versus District 3) 

-.007 

.064 

.915 

D i stri ct 4 ( versus D i stri ct 3) 

.111 

.067 

.103 

School-level covariates 




School-level prior mathematics achievement 11 

.084 

.054 

.123 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.177 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.070 

.029 

.016 

Female 

.021 

.036 

.560 

Student-level covariates 




Prior mathematics achievement 0 

.724 

.020 

<.0001 

Grade 5 (versus Grade 4) 

.008 

.047 

.868 

Hispanic (versus white) 

-.060 

.018 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.087 

.031 

.005 

African American (versus white) 

-.147 

.020 

<.0001 

American Indian (versus white) 

-.047 

.050 

.350 

M ultiracial (versus white) 

1— 1 
O 
"-I 

.090 

.236 

Eligible for free or reduced-price lunch 

o 
1 — 1 
1 — 1 

.016 

<.0001 

Female 

.008 

.012 

.504 

IEP 

-.028 

.018 

.129 

Interaction terms 




District 1 x Grade 5 

-.021 

.065 

.740 

District 2 x Grade 5 

-.042 

.065 

.519 

D istrict 4 x Grade 5 

-.046 

.077 

.552 

District 1 x prior mathematics achievement 

-.038 

.027 

.163 

District 2 x prior mathematics achievement 

-.022 

.025 

.378 

District 4 x prior mathematics achievement 

-.163 

.031 

<.0001 

Grade 5 x prior mathematics achievement 

.187 

.028 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

-.167 

.038 

<.0001 

District 2 x Grade 5 x prior mathematics achievement 

-.130 

.036 

0 

District 4 x Grade 5 x prior mathematics achievement 

-.147 

.045 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.352 

.005 

<.0001 

Level 2 

.055 

.006 

<.0001 

Level 3 

.018 

.005 

.001 

Total variance 

.425 




The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average Of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and standard deviation of test scores 
within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful, efficient way to 
combine results from assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D. 22. Estimated Relationship Between Supports for Data UseAmong 
Elementary School Principals and Student Achievement in M athematics (N=9,266) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

Intercept 

.018 

.048 

.706 

Supports for Data Use a 

.109 

.052 

.036 

District covariates 




District 1 [versus District 3) 

-.003 

.067 

.961 

D i stri ct 2 ( versus D i stri ct 3) 

.006 

.066 

.924 

D i stri ct 4 ( versus D i stri ct 3) 

.085 

.068 

.210 

School-level covariates 




School-level prior mathematics achievement 11 

.099 

.057 

.086 

Teacher-level covariates 




T otal years of teaching experience 

.003 

.002 

.077 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.056 

.030 

.061 

Female 

.035 

.038 

.362 

Student-level covariates 




Prior mathematics achievement 0 

.725 

.021 

<.0001 

Grade 5 [versus Grade 4) 

.007 

.046 

.878 

Hispanic [versus white) 

0.065 

.019 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.087 

.033 

.009 

African American [versus white) 

-.134 

.021 

<.0001 

American Indian [versus white) 

-.045 

.053 

.399 

M ulti racial [versus white) 

-.102 

.091 

.263 

Eligible for free or reduced-price lunch 

-.114 

.017 

<.0001 

Female 

.013 

.012 

.303 

IEP 

-.030 

.019 

.126 

Interaction terms 




District 1 x Grade 5 

-.055 

.066 

.399 

District 2 x Grade 5 

-.032 

.067 

.634 

District 4 x Grade 5 

-.045 

.076 

.554 

District 1 x prior mathematics achievement 

-.047 

.028 

.099 

District 2 x prior mathematics achievement 

-.024 

.026 

.352 

District 4 x prior mathematics achievement 

-.164 

.031 

<.0001 

Grade 5 x prior mathematics achievement 

.189 

.029 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

0.173 

.040 

<.0001 

District 2 x Grade 5 x prior mathematics achievement 

-.128 

.037 

0.001 

District 4 x Grade 5 x prior mathematics achievement 

-.149 

.045 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.355 

.005 

<.0001 

Level 2 

.051 

.006 

<.0001 

Level 3 

.018 

.006 

.001 

Total variance 

.424 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D.23. Estimated Relationship Between Attention to Data in the School Among 
Elementary School Principals and Student Achievement in M athematics (N=9,690) 


Fixed Effect 

Coefficient 

Std Error 

p Value 

1 ntercept 

.023 

.048 

.634 

Working with Data/I nstructional Response 3 

.096 

.045 

.033 

District covariates 




District 1 (versus District 3) 

0 

.067 

.995 

D i stri ct 2 ( versus D i stri ct 3) 

.023 

.067 

.731 

D i stri ct 4 ( versus D i stri ct 3) 

.069 

.070 

.321 

School-level covariates 




School-level prior mathematics achievement 6 

.135 

.058 

.022 

Teacher-level covariates 




T otal years of teaching experience 

.003 

.002 

.049 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.054 

.029 

.062 

Female 

.043 

.036 

.236 

Student-level covariates 




Prior mathematics achievement 6 

.723 

.021 

<.0001 

Grade 5 (versus Grade 4) 

.009 

.046 

.839 

Hispanic (versus white) 

-.062 

.018 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.092 

.031 

.004 

African American (versus white) 

-.144 

.020 

<.0001 

American Indian (versus white) 

-.042 

.051 

.412 

M ultiracial (versus white) 

-.105 

.091 

.248 

Eligible for free or reduced-price lunch 

-.112 

.017 

<.0001 

Female 

.010 

.012 

.431 

IEP 

-.023 

.019 

.216 

Interaction terms 




District 1 x Grade 5 

-.045 

.065 

.482 

District 2 x Grade 5 

-.028 

.065 

.662 

D istrict 4 x Grade 5 

-.048 

.076 

.525 

District 1 x prior mathematics achievement 

-.043 

.028 

.126 

District 2 x prior mathematics achievement 

-.029 

.026 

.262 

District 4 x prior mathematics achievement 

-.163 

.031 

<.0001 

Grade 5 x prior mathematics achievement 

.189 

.029 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

-170 

.039 

<.0001 

District 2 x Grade 5 x prior mathematics achievement 

-.122 

.036 

.001 

D istrict 4 x Grade 5 x prior mathematics achievement 

-.148 

.045 

.001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.353 

.005 

<.0001 

Level 2 

.050 

.005 

<.0001 

Level 3 

.019 

.006 

0 

Total variance 

.422 




The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.24. Estimated Relationship Between BanierstoDataUseAmong 
E lementary School Principals and Student Achievement in M athematics (N=8,373) 


Fixed Effect 

Coefficient 

Std Error 

p Value 

1 ntercept 

.031 

.053 

.562 

Barriers to Data Use 3 

-.031 

.060 

.608 

District covariates 




District 1 (versus District 3) 

.010 

.077 

.898 

D i stri ct 2 ( versus D i stri ct 3) 

-.037 

.069 

.591 

D i stri ct 4 ( versus D i stri ct 3) 

.140 

.072 

.057 

School-level covariates 




School-level prior mathematics achievement 6 

.098 

.059 

.101 

Teacher-level covariates 




T otal years of teaching experience 

.004 

.002 

.031 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.047 

.031 

.128 

Female 

.055 

.039 

.156 

Student-level covariates 




Prior mathematics achievement 0 

.730 

.022 

<.0001 

Grade 5 (versus Grade 4) 

-.015 

.047 

.749 

Hispanic (versus white) 

-.048 

.020 

.017 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.087 

.033 

.008 

African American (versus white) 

-.150 

.021 

<.0001 

American Indian (versus white) 

-.022 

.059 

.707 

M ultiracial (versus white) 

-.147 

.093 

.116 

Eligible for free or reduced-price lunch 

-.110 

.018 

<.0001 

Female 

.012 

.013 

.340 

IEP 

-.035 

.021 

.088 

Interaction terms 




District 1 x Grade 5 

-.018 

.070 

.798 

District 2 x Grade 5 

-.003 

.065 

.961 

D istrict 4 x Grade 5 

-.076 

.077 

.322 

District 1 x prior mathematics achievement 

-.057 

.031 

.068 

District 2 x prior mathematics achievement 

.028 

.027 

.296 

District 4 x prior mathematics achievement 

-.175 

.033 

<.0001 

Grade 5 x prior mathematics achievement 

.192 

.030 

<.0001 

District 1 x Grade 5 x prior mathematics achievement 

-.116 

.043 

.008 

District 2 x Grade 5 x prior mathematics achievement 

-.133 

.037 

0 

D istrict 4 x Grade 5 x prior mathematics achievement 

-.143 

.046 

.002 

Random Effect 

Variance 

Std Error 

p Value 

Level 1 

.346 

.005 

<.0001 

Level 2 

.047 

.005 

<.0001 

Level 3 

.021 

.006 

.001 

Total variance 

.414 




Vhe key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 


Council of the Great City Schools • American Institutes for Research • Summer 2012 


APPENDICES CONT’D 


Table D.25. Estimated Relationship Between ContBctforDataUseAmong 
Elementary School Principals and Student Achievement in Reading (N=10,175) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.002 

.042 

.960 

Context for Data Use a 

.036 

.048 

.447 

District covariates 




D i Stri Ct 1 ( versus D i Stri Ct 3) 

.004 

.059 

.941 

D i Stri Ct 2 ( versus D i Stri Ct 3) 

-.029 

.055 

.602 

D i Stri Ct 4 ( versus D i Stri Ct 3) 

.086 

.057 

.136 

School-level covariates 




School-level prior reading achievement 6 

.104 

.046 

.028 

Teacher-level covariates 




T otal years of teaching experience 

.003 

.001 

.032 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.037 

.026 

.156 

Female 

-.007 

.032 

.839 

Student-level covariates 




Prior reading achievement 6 

.653 

.021 

<.0001 

G rade 5 [versus Grade 4) 

.002 

.045 

.973 

Hispanic ( versus white) 

-.063 

.019 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.016 

.032 

.620 

African American (versus white) 

-.142 

.020 

<.0001 

American Indian (versus white) 

-.240 

.052 

<.0001 

M ulti racial (versus white) 

-.070 

.094 

.456 

Eligiblefor free or reduced-price lunch 

-.095 

.017 

<.0001 

Female 

.131 

.012 

<.0001 

IEP 

-.257 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.017 

.061 

.775 

District 2 x Grade 5 

-.010 

.061 

.865 

D i stri ct 4 x Grade 5 

-.046 

.069 

.505 

District 1 x prior reading achievement 

.033 

.028 

.244 

District 2 x prior reading achievement 

.050 

.026 

.052 

D i stri ct 4 x prior reading achievement 

-.105 

.031 

.001 

Grade 5 x prior reading achievement 

.247 

.029 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.313 

039 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.204 

0.036 

<.0001 

D i stri ct 4 x Grade 5 x prior reading achievement 

-.270 

.045 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.376 

.005 

<.0001 

Level 2 

.041 

.005 

<.0001 

Level 3 

.009 

.004 

.007 

T otal variance 

.426 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D. 26. Estimated Relationship Between Supports for Data UseAmong 
Elementary School Principals and Student Achievement in Reading (N=9,365) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.000 

.041 

.996 

Supports for Data UseP 

.091 

.043 

.036 

District covariates 




District 1 (versus District 3) 

.008 

.058 

.895 

District 2 (versus District 3) 

-.004 

.056 

.940 

District 4 (versus District 3) 

.072 

.058 

.217 

School-level covariates 




School-level prior reading achievement 13 

.124 

.050 

.014 

Teacher-level covariates 




T otal years of teaching experience 

.004 

.001 

.011 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.023 

.026 

.391 

Female 

.011 

.034 

.752 

Student-level covariates 




Prior reading achievement 3 

.646 

.021 

<.0001 

Grade 5 (versus Grade 4) 

.005 

.045 

.908 

Hispanic (versus white) 

-.064 

.019 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.017 

.034 

.625 

African American (versus white) 

-.142 

.021 

<.0001 

American Indian (versus white) 

-.246 

.054 

<.0001 

M ultiracial (versus white) 

-.062 

.094 

.510 

Eligible for free or reduced-price lunch 

-.100 

.017 

<.0001 

Female 

.130 

.013 

<.0001 

IEP 

-.262 

.020 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.037 

.062 

.554 

District 2 x Grade 5 

-.020 

.062 

.753 

District 4 x Grade 5 

-.047 

.068 

.488 

District 1 x prior reading achievement 

.049 

.029 

.094 

District 2 x prior reading achievement 

.058 

.026 

.028 

District 4 x prior reading achievement 

-.098 

.031 

.002 

Grade 5 x prior reading achievement 

.256 

.029 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.344 

.041 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.223 

.037 

<.0001 

District 4 x Grade 5 x prior reading achievement 

-.279 

.045 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.373 

.006 

<.0001 

Level 2 

.038 

.005 

<.0001 

Level 3 

.010 

.004 

.006 

Total variance 

.421 




^The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average Of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D.27. Estimated Relationship Between Attention to Data in the School Among 
Elementary School Principals and Student Achievement in Reading (N=9,788) 


Fixed Effect 

Coefficient 

Std Error 

p Value 

1 ntercept 

.002 

.041 

.963 

Working with Data/I nstructional Response 3 

.007 

.037 

.856 

District covariates 




District 1 ( versus District 3) 

.007 

.058 

.908 

D i stri ct 2 ( versus D i stri ct 3) 

-.021 

.057 

.714 

D i stri ct 4 ( versus D i stri ct 3) 

.085 

.059 

.150 

School-level covariates 




School-level prior reading achievement 13 

.109 

.050 

.031 

Teacher -lev el covariates 




T otal years of teaching experience 

.003 

.001 

.020 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.026 

.026 

.310 

Female 

.008 

.032 

.793 

Student-level covariates 




Prior reading achievement 

.645 

.021 

<.0001 

Grade 5 (verms Grade 4) 

.006 

.044 

.891 

Hispanic (versus white) 

-.062 

.019 

.001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.013 

.032 

.682 

African American (versus white) 

-.143 

.020 

<.0001 

American Indian (versus white) 

-.238 

.052 

<.0001 

M ultiracial (versus white) 

-.062 

.095 

.514 

Eligible for free or reduced-price lunch 

-.096 

.017 

<.0001 

Female 

.130 

.012 

<.0001 

IEP 

-.262 

.019 

<.0001 

Interaction terms 




District 1 x Grade 5 

-.028 

.061 

.639 

District 2 x Grade 5 

-.018 

.060 

.762 

District 4 x Grade 5 

-.047 

.067 

.483 

District 1 x prior reading achievement 

.046 

.029 

.113 

District 2 x prior reading achievement 

.054 

.026 

.037 

District 4 x prior reading achievement 

-.097 

.031 

.002 

Grade 5 x prior reading achievement 

.255 

.029 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.333 

.040 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.216 

.037 

<.0001 

District 4 x Grade 5 x prior reading achievement 

-.278 

.045 

<.0001 

Random Effect 

Variance 

Std Error 

p Value 

Level 1 

.374 

.006 

<.0001 

Level 2 

.037 

.005 

<.0001 

Level 3 

.010 

.004 

.004 

Total variance 

.422 




‘The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.28. Estimated Relationship Between BamerstoDatallseAmong 
Elementary School Principals and Student Achievement in Reading (N=8,430) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.009 

.045 

.845 

Barriers to Data Use 3 

-.037 

.049 

.448 

District covariates 




District 1 (versus District 3) 

-.030 

.064 

.647 

D i stri ct 2 ( versus D i stri ct 3) 

-.023 

.058 

.689 

D i stri ct 4 ( versus D i stri ct 3) 

.066 

.061 

.277 

School-level covariates 




School-level prior reading achievement 11 

.108 

.050 

.034 

Teacher-level covariates 




T otal years of teaching experience 

.004 

.002 

.012 

Education above a bachelor’s degree ( versus bachelor’s degree) 

.030 

.027 

.277 

Female 

.008 

.034 

.807 

Student-level covariates 




Prior reading achievement 0 

.675 

.023 

<.0001 

Grade 5 (versus Grade 4) 

-.010 

.047 

.825 

Hispanic (versus white) 

-.058 

.021 

.005 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

-.004 

.034 

.899 

African American (versus white) 

-.129 

.022 

<.0001 

American Indian (versus white) 

-.250 

.061 

<.0001 

M ultiracial (versus white) 

-.100 

.099 

.316 

Eligible for free or reduced-price lunch 

-.113 

.019 

<.0001 

Female 

.123 

.013 

<.0001 

IEP 

-.249 

.022 

<.0001 

Interaction terms 




District 1 x Grade 5 

.015 

.067 

.820 

District 2 x Grade 5 

-.014 

.062 

.827 

D istrict 4 x Grade 5 

-.041 

.070 

.553 

District 1 x prior reading achievement 

.013 

.033 

.693 

District 2 x prior reading achievement 

.022 

.028 

.418 

D istrict 4 x prior reading achievement 

-.136 

.033 

<.0001 

Grade 5 x prior reading achievement 

.234 

.031 

<.0001 

District 1 x Grade 5 x prior reading achievement 

-.261 

.046 

<.0001 

District 2 x Grade 5 x prior reading achievement 

-.185 

.038 

<.0001 

D istrict 4 x Grade 5 x prior reading achievement 

-.252 

.047 

<.0001 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.374 

006 

<.0001 

Level 2 

.035 

.005 

<.0001 

Level 3 

.010 

.004 

.007 

Total variance 

.419 




The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 
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Table D.29. Estimated Relationship Between CaihextfbrDataUseAmong 
M iddle School Principals and Student Achievement in M athematics (N=26,708) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

-.014 

.046 

.758 

Context for Data Use a 

.031 

.048 

.518 

District covariates 




D i Stri Ct 1 ( versus D i Stri Ct 3) 

i_n 

i — i 

o 

.057 

.799 

D i Stri Ct 2 ( versus D i Stri Ct 3) 

.002 

.054 

.977 

D i Stri Ct 4 ( versus D i Stri Ct 3) 

.282 

.083 

.001 

School-level covariates 




School-level prior mathematics achievement* 

.220 

.050 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.269 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.027 

.037 

.466 

Female 

.050 

.037 

.174 

Student-level covariates 




Prior mathematics achievement 0 

.760 

.012 

<.0001 

Grade 8 (versus Grade 7) 

.048 

.052 

.360 

H ispanic ( versus white) 

o 

i — i 

.011 

<.0001 

Asian/Native Hawaiian /Pacific Islander (versus white) 

.093 

.016 

<.0001 

African American (versus white) 

-.127 

.012 

<.0001 

American Indian (versus white) 

o 

i — i 

.035 

.003 

M ulti racial (versus white) 

-.019 

.080 

0.810 

Eligible for free or reduced-price lunch 

-.090 

.010 

<.0001 

Female 

-.033 

.007 

<.0001 

IEP 

.002 

.014 

0.895 

Interaction terms 




District 1 x Grade 8 

-.248 

.060 

<.0001 

District 2 x Grade 8 

-.079 

.056 

0.159 

D i stri ct 4 x Grade 8 

-.082 

.082 

0.315 

District 1 x prior mathematics achievement 

CO 

I — 1 

m 

.017 

<.0001 

District 2 x prior mathematics achievement 

-.011 

.015 

0.453 

District 4 x prior mathematics achievement 

-.200 

.031 

<.0001 

G rade 8 x prior mathematics achievement 

.054 

.018 

0.002 

District 1 x Grade 8 x prior mathematics achievement 

-.099 

.024 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.120 

.021 

<.0001 

D i stri ct 4 x Grade 8 x prior mathematics achievement 

-.110 

.049 

0.025 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.086 

.008 

<.0001 

Level 3 

0 

0 

0 

T otal variance 

.433 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.30. Estimated Relationship Between SupportsforDatallseAmong 
M iddle School Principals and Student Achievement in M athematics (N=23,241) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

-.028 

.050 

.569 

Supports for Data Use 3 

.062 

.055 

.261 

District covariates 




District 1 ( versus District 3) 

.000 

.062 

.997 

District 2 ( versus District 3) 

.018 

.062 

.769 

D i Stri Ct 4 ( versus D i Stri Ct 3) 

.303 

.088 

.001 

School-level covariates 




School-level prior mathematics achievement 11 

.223 

.055 

0 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.310 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.015 

.041 

.715 

Female 

.047 

.041 

.251 

Student-level covariates 




Prior mathematics achievement 

.767 

.013 

<.0001 

Grade 8 (versus Grade 7) 

.047 

.055 

.393 

Hispanic ( versus white) 

-.108 

.012 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.110 

.018 

<.0001 

African American ( versus white) 

-.138 

.013 

<.0001 

American Indian (versus white) 

-.103 

.037 

.005 

M ultiracial (versus white) 

-.042 

.086 

.621 

Eligible for free or reduced-price lunch 

-.100 

.011 

<.0001 

Female 

-.034 

.008 

<.0001 

IEP 

.029 

.015 

.047 

Interaction terms 




District 1 x Grade 8 

-.252 

.063 

<.0001 

District 2 x Grade 8 

-.097 

.061 

.108 

District 4 x Grade 8 

-.086 

.085 

.311 

District 1 x prior mathematics achievement 

-.327 

.018 

<.0001 

District 2 x prior mathematics achievement 

-.016 

.016 

.316 

District 4 x prior mathematics achievement 

-.209 

.031 

<.0001 

Grade 8 x prior mathematics achievement 

.055 

.019 

.003 

District 1 x Grade 8 x prior mathematics achievement 

-.100 

.025 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.120 

.023 

<.0001 

District 4 x Grade 8 x prior mathematics achievement 

-.108 

.050 

.030 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.353 

.003 

<.0001 

Level 2 

.096 

.009 

<.0001 

Level 3 

0 

0 

0 

Total variance 

.449 




The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 
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Table D.31. Estimated Relationship Between Attention to Data in the School Among 
M iddle School Principals and Student Achievement in M athematics (N=26,085) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

-.014 

.046 

.765 

Working with Data/I nstructional Response 3 

.040 

.037 

.283 

District covariates 




D i stri ct 1 ( versus D i stri ct 3) 

-.015 

.057 

.787 

D i stri ct 2 ( versus D i stri ct 3) 

.018 

.056 

.753 

D i stri ct 4 ( versus D i stri ct 3) 

.266 

.085 

.003 

School-level covariates 




School-level prior mathematics achievement 13 

.218 

.051 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

.002 

.002 

.329 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.025 

.038 

.509 

Female 

.050 

.038 

.185 

Student-level covariates 




Prior mathematics achievement 13 

.760 

.012 

<.0001 

Grade 8 (versus Grade 7) 

.048 

.053 

.365 

Hispanic (versus white) 

-.106 

.011 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.098 

.017 

<.0001 

African American (versus white) 

-.129 

.012 

<.0001 

American Indian (versus white) 

-.110 

.035 

.002 

M ultiracial (versus white) 

-.020 

.080 

.805 

Eligible for free or reduced-price lunch 

-.095 

.010 

<.0001 

Female 

-.032 

.007 

<.0001 

IEP 

.008 

.014 

.585 

Interaction terms 




District 1 x Grade 8 

-.250 

.061 

<.0001 

District 2 x Grade 8 

-.091 

.057 

.109 

District 4 x Grade 8 

-.083 

.082 

.313 

District 1 x prior mathematics achievement 

-.318 

.017 

<.0001 

District 2 x prior mathematics achievement 

-.014 

.015 

.352 

District 4 x prior mathematics achievement 

-.200 

.031 

<.0001 

Grade 8 x prior mathematics achievement 

.054 

.018 

.002 

District 1 x Grade 8 x prior mathematics achievement 

-.099 

.024 

<.0001 

District 2 x Grade 8 x prior mathematics achievement 

-.120 

.021 

<.0001 

D i strict 4 x Grade 8 x prior mathematics achievement 

-.109 

.049 

.026 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.088 

.008 

<.0001 

Level 3 

0 

0 

0 

Total variance 

.435 




The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.32. Estimated Relationship Between BarrierstoDalaUseAmong 
M iddle School Principals and Student Achievement in M athematics (N=20,931) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

-.005 

.056 

.935 

Barriers to Data Use 3 

-.021 

.055 

.708 

District covariates 




District 1 ( versus District 3) 

-.030 

.066 

.656 

D i stri ct 2 ( versus D i stri ct 3) 

.023 

.065 

.725 

D i stri ct 4 ( versus D i stri ct 3) 

.310 

.101 

.003 

School-level covariates 




School-level prior mathematics achievement 

.153 

.062 

.018 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.002 

.584 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.022 

.046 

.626 

Female 

.053 

.045 

.240 

Student-level covariates 




Prior mathematics achievement 

.780 

.014 

<.0001 

G rade 8 (versus Grade 7) 

.078 

.060 

.192 

Hispanic ( versus white) 

-.094 

.012 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.093 

.018 

<.0001 

African American (versus white) 

-.129 

.014 

<.0001 

American Indian (versus white) 

-.104 

.038 

.006 

M ulti racial (versus white) 

-.064 

.093 

.490 

Eligible for free or reduced-price lunch 

-.102 

.011 

<.0001 

Female 

-.027 

.008 

.001 

IEP 

-.003 

.016 

.832 

Interaction terms 




District 1 x Grade 8 

-.323 

.068 

<.0001 

District 2 x Grade 8 

O 

o 

I — 1 

.064 

.118 

D i stri ct 4 x Grade 8 

I — 1 

LO 

I — 1 

.094 

.108 

District 1 x prior mathematics achievement 

-.392 

.019 

<.0001 

District 2 x prior mathematics achievement 

-.032 

.017 

.059 

D i stri ct 4 x pri or mathemati cs achi evement 

-.231 

.034 

<.0001 

Grade 8 x prior mathematics achievement 

.045 

.019 

.020 

District 1 x Grade 8 x prior mathematics achievement 

-.055 

.026 

.036 

District 2 x Grade 8 x prior mathematics achievement 

-.116 

.023 

<.0001 

D i stri ct 4 x Grade 8 x prior mathematics achievement 

-.127 

.053 

.016 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.339 

.003 

<.0001 

Level 2 

.097 

.010 

<.0001 

Level 3 

0 

0 

0 

T otal variance 

.436 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state mathematics assessment, because all four participating districts use different state tests, it was necessary to translate the 
scores into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test 
scores within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient 
way to combine results from different assessments (M ay et al., 2009). 
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APPENDICES CONT’D 


Table D.33. Estimated Relationship Between ContBctforDataUseAmong 
M iddle School Principals and Student Achievement in Reading (N=26,632) 


Fixed Effect 

Coefficient 

Std E rror 

p value 

1 ntercept 

.078 

.049 

.112 

Context for Data Use 3 

-.064 

.056 

.257 

District covariates 




District 1 (versus District 3) 

-.119 

.064 

.069 

D i stri ct 2 ( versus D i stri ct 3) 

-.140 

.060 

.023 

D i stri ct 4 ( versus D i stri ct 3) 

-.032 

.086 

.713 

School-level covariates 




School-level prior reading achievement 6 

.313 

.055 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

.000 

.002 

.912 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.024 

.038 

.535 

Female 

.135 

.045 

.003 

Student-level covariates 




Prior reading achievement 6 

.708 

.013 

<.0001 

G rade 8 (verms G rade 7) 

-.042 

.052 

.416 

Hispanic (versus white) 

-.075 

.011 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.003 

.017 

.868 

African American (versus white) 

-.131 

.013 

<.0001 

American Indian (versus white) 

-.146 

.037 

<.0001 

M ultiracial (versus white) 

-.012 

.084 

.886 

Eligible for free or reduced-price lunch 

-.093 

.010 

<.0001 

Female 

.114 

.008 

<.0001 

IEP 

-.239 

.016 

<.0001 

Interaction Terms 




District 1 x Grade 8 

-.044 

.061 

.467 

District 2 x Grade 8 

.101 

.058 

.082 

District 4 x Grade 8 

.063 

.077 

.414 

District 1 x prior reading achievement 

-.236 

.019 

<.0001 

District 2 x prior reading achievement 

0014 

.016 

.390 

District 4 x prior reading achievement 

-.089 

.030 

.003 

Grade 8 x prior reading achievement 

-.015 

.020 

.465 

District 1 x Grade 8 x prior reading achievement 

-.025 

.027 

.358 

District 2 x Grade 8 x prior reading achievement 

.027 

.023 

.244 

District 4 x Grade 8 x prior reading achievement 

-.049 

.041 

.232 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.078 

.008 

<.0001 

Level 3 

.009 

.004 

.018 

T otal variance 

.434 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 


CHARTING SUCCESS: DATA USE AND STUDENT ACHIEVEMENT IN URBAN SCHOOLS 


Table D.34. Estimated Relationship Between SupportsforDalaUseAmong 
M iddle School Principals and Student Achievement in Reading (N=21,686) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.062 

.052 

.239 

Supports for Data Use a 

.007 

.060 

.912 

District covariates 




District 1 ( versus District 3) 

-.085 

.067 

.212 

District 2 ( versus District 3) 

-.127 

.066 

.059 

D i stri ct 4 ( versus D i stri ct 3) 

-.006 

.089 

.942 

School-level covariates 




School-level prior reading achievement 13 

.345 

.061 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

0 

.002 

.938 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.030 

.041 

.453 

Female 

.151 

.049 

.002 

Student-level covariates 




Prior reading achievement 0 

.705 

.013 

<.0001 

G rade 8 [versus Grade 7) 

-.029 

.054 

.588 

Hispanic ( versus white) 

-.079 

.012 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.011 

.017 

.532 

African American (versus white) 

-.141 

.014 

<.0001 

American Indian (versus white) 

-.130 

.039 

.001 

Multiracial (versus white) 

-.052 

.088 

.559 

Eligible for free or reduced-price lunch 

-.098 

.011 

<.0001 

Female 

.120 

.008 

<.0001 

IEP 

-.244 

.016 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.060 

.063 

.343 

District 2 x Grade 8 

.079 

.061 

.193 

District 4 x Grade 8 

.051 

.079 

.518 

District 1 x prior reading achievement 

-.236 

.019 

<.0001 

District 2 x prior reading achievement 

.015 

.017 

.358 

District 4 x prior reading achievement 

-.089 

.030 

.003 

Grade 8 x prior reading achievement 

-.009 

.020 

.673 

District 1 x Grade 8 x prior reading achievement 

-.030 

.028 

.270 

District 2 x Grade 8 x prior reading achievement 

.030 

.024 

.215 

District 4 x Grade 8 x prior reading achievement 

-.055 

.041 

.184 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.347 

.003 

<.0001 

Level 2 

.084 

.009 

<.0001 

Level 3 

.010 

.005 

.020 

Total variance 

.440 




‘“The key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al 2009). 
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APPENDICES CONT’D 


Table D.35. Estimated Relationship Between Attention to Data in the School Among 
Middle School Principals and Student Achievement in Reading (N=23,809) 


Fixed Effect 

C oefficient 

Std E rror 

p Value 

1 ntercept 

.075 

.049 

.130 

Working with Data/I nstructional Response 3 

.005 

.045 

.915 

District covariates 




District 1 (versus District 3) 

-.099 

.063 

.122 

D i stri ct 2 ( versus D i stri ct 3) 

-.142 

.061 

.024 

D i stri ct 4 ( versus D i stri ct 3) 

-.025 

.088 

.775 

School-level covariates 




School-level prior reading achievement 13 

.312 

.056 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

.000 

.002 

.999 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.024 

.038 

.517 

Female 

.139 

.045 

.002 

Student-level covariates 




Prior reading achievement 

.707 

.013 

<.0001 

Grade 8 ( versus Grade 7) 

-.043 

.052 

.408 

Hispanic (versus white) 

-.080 

.011 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.004 

.016 

.808 

African American (versus white) 

-.135 

.013 

<.0001 

American Indian (versus white) 

-.152 

.037 

<.0001 

M ultiracial (versus white) 

-.014 

.083 

.871 

Eligible for free or reduced-price lunch 

-.092 

.010 

<.0001 

Female 

.117 

.008 

<.0001 

IEP 

-.242 

.016 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.044 

.061 

.470 

District 2 x Grade 8 

.104 

.058 

.073 

D istrict 4 x Grade 8 

.064 

.077 

.405 

District 1 x prior reading achievement 

-.235 

.019 

<.0001 

District 2 x prior reading achievement 

.017 

.016 

.271 

District 4 x prior reading achievement 

-.089 

.030 

.003 

Grade 8 x prior reading achievement 

-.014 

.020 

.466 

District 1 x Grade 8 x prior reading achievement 

-.025 

.027 

.356 

District 2 x Grade 8 x prior reading achievement 

.033 

.023 

.159 

District 4 x Grade 8 x prior reading achievement 

-.049 

.041 

.230 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.345 

.003 

<.0001 

Level 2 

.078 

.008 

<.0001 

Level 3 

.010 

.004 

.014 

Total variance 

.433 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 
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Table D. 36. Estimated Relationship Between BarrierstoDalaUseAmong 
M iddle School Principals and Student Achievement in Reading (N=18 f 348) 


Fixed Effect 

Coefficient 

Std E rror 

p Value 

1 ntercept 

.043 

.059 

.472 

Barriers to Data Use a 

.104 

.064 

.104 

District covariates 




D i Stri Ct 1 (versus D i Stri Ct 3) 

-.157 

.072 

.033 

D i Stri Ct 2 (versus D i Stri Ct 3) 

-.154 

.071 

.034 

D i Stri Ct 4 (versus D i Stri Ct 3) 

.046 

.100 

.646 

School-level covariates 




School-level prior reading achievement 11 

.288 

.068 

<.0001 

Teacher-level covariates 




T otal years of teaching experience 

.001 

.002 

.642 

Education above a bachelor’s degree ( versus bachelor’s degree) 

-.021 

.048 

.658 

Female 

.168 

.058 

.004 

Student-level covariates 




Prior reading achievement 

.720 

.014 

<.0001 

Grade 8 (verm? Grade 7) 

-.061 

.062 

.319 

Hispanic (versus white) 

-.085 

.013 

<.0001 

Asian/Native Hawaiian /Pacific Islander ( versus white) 

.011 

.018 

.559 

African American (versus white) 

-.130 

.015 

<.0001 

American Indian (versus white) 

-.143 

.040 

.000 

M ulti racial (versus white) 

.015 

.098 

.882 

E ligible for free or reduced- price lunch 

-.093 

.012 

<.0001 

Female 

.114 

.009 

<.0001 

IEP 

-.238 

.018 

<.0001 

Interaction terms 




District 1 x Grade 8 

-.008 

.071 

.906 

District 2 x Grade 8 

.169 

.069 

.015 

D i stri ct 4 x Grade 8 

.082 

.086 

.340 

District 1 x prior reading achievement 

-.268 

.021 

<.0001 

District 2 x prior reading achievement 

-.017 

.018 

.342 

D i stri ct 4 x prior reading achievement 

-.120 

.033 

.000 

Grade 8 x prior reading achievement 

-.023 

.022 

.292 

District 1 x Grade 8 x prior reading achievement 

-.036 

.030 

.224 

District 2 x Grade 8 x prior reading achievement 

.049 

.026 

.063 

D i stri ct 4 x Grade 8 x prior reading achievement 

-.014 

.044 

.753 

Random Effect 

Variance 

Std E rror 

p Value 

Level 1 

.343 

.004 

<.0001 

Level 2 

.099 

.011 

<.0001 

Level 3 

.005 

.005 

.154 

T otal variance 

.447 




^he key dimensions of data use for principals were group-mean centered. b Values are standardized school-level aggregates of 
students’ prior achievement. Each school has one unique value for the average of its students’ prior achievement from the 2009 
state reading assessment, because all four participating districts use different state tests, it was necessary to translate the scores 
into a common metric. All scores were standardized to Z-scores using the mean and the standard deviation of the test scores 
within each district and grade. Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to 
combine results from different assessments (M ay et al., 2009). 
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